{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# ニューラルネットワーク\n",
    "\n",
    "## パーセプトロンからニューラルネットワークへ\n",
    "\n",
    "ニューラルネットワークの図\n",
    "\n",
    "<img src=\"img/3-1.gif\" alt=\"ニューラルネットワークの例\" title=\"ニューラルネットワークの例\" width=\"300\" height=\"300\" />  \n",
    "\n",
    "* 入力層\n",
    "    * 左の列\n",
    "    * ここでは第0層とする\n",
    "* 中間層\n",
    "    * 中間の列\n",
    "    * ここでは第1層とする\n",
    "    * 隠れ層とも呼ばれる\n",
    "        * 入力層や出力層とちがって、人の目には見えない層だから\n",
    "* 出力層\n",
    "    * 一番右の列\n",
    "    * ここでは第2層とする\n",
    "\n",
    "---\n",
    "\n",
    "パーセプトロンの式\n",
    "\n",
    "$\n",
    "y = \n",
    "\\begin{eqnarray}\n",
    "\\left\\{\n",
    "\\begin{array}{l}\n",
    "0 & ( b + \\omega_{1} x_{1} + \\omega_{2} x_{2} \\leqq 0 ) \\\\\n",
    "1 & ( b + \\omega_{1} x_{1} + \\omega_{2} x_{2} > 0 ) \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\end{eqnarray}\n",
    "$\n",
    "\n",
    "でバイアスを明示的に図示すると以下\n",
    "   \n",
    "<img src=\"img/3-3.png\" alt=\"バイアスを明示的に示す\" title=\"バイアスを明示的に示す\" width=\"300\" height=\"300\" />  \n",
    "\n",
    "* 重みが $ b $ で入力が1の信号が追加されている\n",
    "    * **バイアスの信号は常に1**であるため、図で表す際にはニューロンを灰色で塗りつぶして他のニューロンと差別化する\n",
    "\n",
    "式をよりシンプルな形に書き換える\n",
    "\n",
    "$\n",
    "y = h ( b + \\omega_{1} x_{1} + \\omega_{2} x_{2} )  \\\\\n",
    "h(x) = \n",
    "\\begin{eqnarray}\n",
    "\\left\\{\n",
    "\\begin{array}{l}\n",
    "0 & ( x \\leqq 0 ) \\\\\n",
    "1 & ( x > 0 ) \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\end{eqnarray}\n",
    "$\n",
    "\n",
    "* 場合分け(0を超えたら1を出力、そうでなければ0を出力)を一つの関数 $ h(x) $ で表現\n",
    "    * これを**活性化関数(activation function)**という\n",
    "        * 入力信号の総和を出力信号に変換する関数\n",
    "\n",
    "さらに式を置き換えると\n",
    "\n",
    "$\n",
    "a = b + \\omega_{1} x_{1} + \\omega_{2} x_{2} \\\\\n",
    "y = h(a)\n",
    "$\n",
    "\n",
    "* a : 重み付き入力信号とバイアスの総和\n",
    "\n",
    "これを図示すると以下になる\n",
    "\n",
    "<img src=\"img/3-4.png\" alt=\"活性化関数によるプロセスを明示化した図\" title=\"活性化関数によるプロセスを明示化した図\" width=\"300\" height=\"300\" />  \n",
    "\n",
    "* ニューロンの○の中に活性化関数によるプロセスを明示的に図示\n",
    "    * 重み付き信号の和の結果が $ a $ というノードになる\n",
    "    * 活性化関数 $ h() $ によって $ y $ というノードに変換される\n",
    "        * ここでは「ニューロン」と「ノード」という用語は同じ意味で使う\n",
    "\n",
    "---\n",
    "\n",
    "* パーセプトロンとニューラルネットワークの違い\n",
    "    * ニューロンのつながり方はパーセプトロンと同じ\n",
    "    * 「単純パーセプトロン」のモデル\n",
    "        * 単層のネットワーク\n",
    "        * 活性化関数がステップ関数(階段関数)\n",
    "    * 「多層パーセプトロン」のモデル\n",
    "        * 多層のネットワーク\n",
    "        * 活性化関数がステップ関数\n",
    "    * 「ニューラルネットワーク」のモデル\n",
    "        * 多層ネットワーク\n",
    "        * 活性化関数がシグモイド関数など**滑らかな活性化関数を使用**\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 活性化関数\n",
    "\n",
    "* 非線形関数である\n",
    "    * 線形関数は使えない\n",
    "        * 線形関数とは\n",
    "            * **出力が入力の定数倍になるような関数**が線形関数\n",
    "            * 真っ直ぐな一本の直線になる\n",
    "        * なぜ使えないか\n",
    "            * ニューラルネットワークで層を深くする事の意味が無くなる\n",
    "                * どんなに層を深くしても、それと同じことを行う「隠れ層の無いネットワーク」が必ず存在するため\n",
    "                * $ h(x) = cx, y(x) = h(h(h(x))), y(x) = c^3x $ <- 一回の掛け算で表現できてしまう\n",
    "                * -> 隠れ層のないネットワークで表現できてしまう\n",
    "    * 非線形関数\n",
    "        * ステップ関数\n",
    "            * 階段のような折れ曲がった直線\n",
    "        * シグモイド関数\n",
    "            * 曲線\n",
    "            \n",
    "    \n",
    "    \n",
    "\n",
    "### ステップ関数\n",
    "\n",
    "$\n",
    "h(x) = \n",
    "\\begin{eqnarray}\n",
    "\\left\\{\n",
    "\\begin{array}{l}\n",
    "0 & ( x \\leqq 0 ) \\\\\n",
    "1 & ( x > 0 ) \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\end{eqnarray}\n",
    "$\n",
    "\n",
    "* 式中の $ x $ は実数を取ることができない\n",
    "    * NumPy配列に対応していない"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEVVJREFUeJzt3X+MHOddx/HPx3cOoSRN1PgQ4LNzprgSVlKU6uRG5I9G\nJCA7FBsJimIUoBDV/9QoVQPIJSitUiRUIlqEaigWVC1tqTHh14k6cgsEVQIS+dL8EHbq6mTS+kxR\n3DRNkdLgnZkvf+zeeXPZmd2cZ3f6jN8vKdLt7pPb7yrPfjL3nWeecUQIANAuG5ouAABQP8IdAFqI\ncAeAFiLcAaCFCHcAaCHCHQBaiHAHgBYi3AGghQh3AGih6abeeNOmTTE3N9fU2wNAkh5//PFvRMTM\nsHGNhfvc3JwWFxebensASJLtr44yjrYMALQQ4Q4ALUS4A0ALEe4A0EKEOwC0EOEOAC1EuANACxHu\nANBChDsAtBDhDgAtRLgDQAsR7gDQQoQ7ALTQ0HC3/XHbz9n+z5LXbfuPbC/Zftr2W+ovEwDwWoxy\n5P4JSbsqXt8taXvvn/2S/uTSywIAXIqh+7lHxBdtz1UM2SvpLyIiJD1q+1rbPxgRX6+pRqBRL77U\n0dPnvtV0GWiRN85cpR+69nvH+h513Kxjs6SzfY+Xe8+9Ktxt71f36F5bt26t4a2B8fvg507poceX\nmy4DLfK7P3uD7rr5+rG+x0TvxBQRhyUdlqT5+fmY5HsD6/Xt73R0/XWv0x+848eaLgUtsfW61439\nPeoI93OStvQ9nu09B7RCXoSuvnJa83NvaLoUYGR1LIVckPTLvVUzN0t6kX472qRThKY2sGoYaRl6\n5G77s5JulbTJ9rKk90vaKEkR8TFJxyTdIWlJ0kuSfnVcxQJNyItCGze46TKA12SU1TL7hrwekt5d\nW0XAd5lOHpoi3JEY/tYEhsiL0PQU4Y60EO7AEFkRmqbnjsQwY4EhsrzQNG0ZJIZwB4bIC3ruSA/h\nDgyRFaGNU3xVkBZmLDBElhccuSM5hDswRPeEKuGOtBDuwBAshUSKCHdgiO5FTHxVkBZmLDBEXrAU\nEukh3IEhMtoySBDhDgyR5ZxQRXoId2CInC1/kSBmLDBEVhTaSFsGiSHcgQpFESpCXMSE5BDuQIWs\n6N7ql547UkO4AxXylXBnbxkkhhkLVOgUhSSO3JEewh2okOfdI3d67kgN4Q5UyGjLIFHMWKBCRlsG\niSLcgQoZbRkkinAHKqysluEiJqSGcAcqrLRl2H4AqWHGAhW4iAmpItyBCis9d8IdqSHcgQoXl0IS\n7kjLSOFue5ft07aXbB8c8PpW24/YfsL207bvqL9UYPLy1aWQHAchLUNnrO0pSYck7Za0Q9I+2zvW\nDPsdSUcj4iZJd0r647oLBZrQoS2DRI1yOLJT0lJEnImIC5KOSNq7ZkxIen3v52sk/Xd9JQLNWVkK\nyTp3pGZ6hDGbJZ3te7ws6a1rxnxA0udt/7qk75N0ey3VAQ1j+wGkqq4Zu0/SJyJiVtIdkj5l+1W/\n2/Z+24u2F8+fP1/TWwPjk+VsP4A0jRLu5yRt6Xs823uu392SjkpSRPyHpCslbVr7iyLicETMR8T8\nzMzM+ioGJiijLYNEjRLuJyRtt73N9hXqnjBdWDPma5JukyTbP6puuHNojuRd3H6AtgzSMnTGRkQm\n6YCk45KeUXdVzEnbD9je0xt2r6R32X5K0mclvTMiYlxFA5PSyVe2H+DIHWkZ5YSqIuKYpGNrnru/\n7+dTkm6ptzSgeTnbDyBR/K0JVOAKVaSKcAcqXNxbhq8K0sKMBSrkBT13pIlwBypk3KwDiSLcgQrc\nZg+pItyBChdv1sFXBWlhxgIVVrf8pS2DxBDuQIWVLX+nTLgjLYQ7UCEvQhssbaDnjsQQ7kCFrAi2\n+0WSmLVAhSwv2HoASSLcgQpZESyDRJIId6BCXgTb/SJJzFqgQlYUHLkjSYQ7UCHLg547kkS4AxXy\nIriACUki3IEKnSLYegBJYtYCFXJ67kgU4Q5UoOeOVBHuQIWMnjsSRbgDFTJ67kgUsxaowPYDSBXh\nDlRg+wGkinAHKrD9AFLFrAUqZDlLIZEmwh2o0D2hSrgjPYQ7UIHtB5CqkcLd9i7bp20v2T5YMuYX\nbJ+yfdL2X9ZbJtCMTl6wFBJJmh42wPaUpEOSflLSsqQTthci4lTfmO2S3ifploh4wfb3j6tgYJJy\nVssgUaMckuyUtBQRZyLigqQjkvauGfMuSYci4gVJiojn6i0TaAZXqCJVo4T7Zkln+x4v957r9yZJ\nb7L9b7Yftb2rrgKBJrG3DFI1tC3zGn7Pdkm3SpqV9EXbN0bEt/oH2d4vab8kbd26taa3BsanexET\nPXekZ5RZe07Slr7Hs73n+i1LWoiITkT8l6SvqBv2rxARhyNiPiLmZ2Zm1lszMDF5UWgjbRkkaJRw\nPyFpu+1ttq+QdKekhTVj/l7do3bZ3qRum+ZMjXUCjchyTqgiTUPDPSIySQckHZf0jKSjEXHS9gO2\n9/SGHZf0vO1Tkh6R9JsR8fy4igYmhYuYkKqReu4RcUzSsTXP3d/3c0h6b+8foDW6FzHRc0d6mLVA\nhU7Blr9IE+EOlCiKUITouSNJhDtQIitCktjyF0li1gIlsqKQxJE70kS4AyVWjtzpuSNFhDtQIs8J\nd6SLcAdKdFbaMvTckSBmLVAipy2DhBHuQImMtgwSRrgDJVZPqLJxGBJEuAMl8tWlkHxNkB5mLVBi\n9SIm2jJIEOEOlFjpuXMRE1JEuAMl6LkjZYQ7UGKl5z5Nzx0JYtYCJToshUTCCHegxOpFTFyhigQx\na4ESnZxdIZEuwh0owfYDSBnhDpRgtQxSRrgDJS7uLcPXBOlh1gIluBMTUka4AyXy1XuoEu5ID+EO\nlGD7AaSMcAdKXLyHKl8TpIdZC5RY3X6AtgwSRLgDJdh+ACkj3IESKydU6bkjRSOFu+1dtk/bXrJ9\nsGLcz9kO2/P1lQg0Y/VmHewtgwQNnbW2pyQdkrRb0g5J+2zvGDDuakn3SHqs7iKBJmTsLYOEjXJI\nslPSUkSciYgLko5I2jtg3AclfUjSyzXWBzQmY28ZJGyUcN8s6Wzf4+Xec6tsv0XSloj4XNUvsr3f\n9qLtxfPnz7/mYoFJyovQ1AbLJtyRnktuJtreIOnDku4dNjYiDkfEfETMz8zMXOpbA2PVKQpaMkjW\nKOF+TtKWvsezvedWXC3pBkn/avtZSTdLWuCkKlKX50FLBskaJdxPSNpue5vtKyTdKWlh5cWIeDEi\nNkXEXETMSXpU0p6IWBxLxcCEZAXhjnQNDfeIyCQdkHRc0jOSjkbESdsP2N4z7gKBpmRFwS32kKzp\nUQZFxDFJx9Y8d3/J2FsvvSygeSsnVIEUcVgClMjy0EbCHYki3IESWRGaYtMwJIpwB0p0T6jyFUGa\nmLlAibwoWC2DZBHuQIlOzglVpItwB0rkRXCjDiSLcAdK0HNHypi5QIksp+eOdBHuQImMtgwSRrgD\nJbpH7nxFkCZmLlCC7QeQMsIdKJEVoY20ZZAowh0okbHOHQkj3IESWUHPHeli5gIluIgJKSPcgRJs\nP4CUEe5AiZzb7CFhhDtQonsRE18RpImZC5TI2PIXCSPcgRI5PXckjHAHSnQvYuIrgjQxc4ESWVFw\n5I5kEe5AiYzVMkgY4Q4MUBShCHGFKpLFzAUG6BSFJHGFKpJFuAMD5EVIEj13JItwBwbIeuFOzx2p\nGincbe+yfdr2ku2DA15/r+1Ttp+2/c+2r6+/VGByspxwR9qGhrvtKUmHJO2WtEPSPts71gx7QtJ8\nRLxZ0kOSfr/uQoFJyno99ynWuSNRo8zcnZKWIuJMRFyQdETS3v4BEfFIRLzUe/iopNl6ywQma6Xn\nvpEjdyRqlHDfLOls3+Pl3nNl7pb08KAXbO+3vWh78fz586NXCUzYSluGE6pIVa1/c9q+S9K8pAcH\nvR4RhyNiPiLmZ2Zm6nxroFarJ1RZColETY8w5pykLX2PZ3vPvYLt2yXdJ+ltEfF/9ZQHNCNfWefO\nRUxI1Cgz94Sk7ba32b5C0p2SFvoH2L5J0p9K2hMRz9VfJjBZHVbLIHFDwz0iMkkHJB2X9IykoxFx\n0vYDtvf0hj0o6SpJf237SdsLJb8OSAIXMSF1o7RlFBHHJB1b89z9fT/fXnNdQKNWeu5s+YtUMXOB\nAbK8t86dI3ckinAHBmC1DFJHuAMDXNx+gK8I0sTMBQZY3X6AtgwSRbgDA6xuP0BbBoki3IEBOmw/\ngMQR7sAAeUHPHWlj5gIDZNxmD4kj3IEBuFkHUke4AwOw/QBSR7gDA7D9AFLHzAUGYJ07Uke4AwPQ\nc0fqCHdggNWlkLRlkChmLjBAZ/VOTBy5I02EOzBAzhWqSBzhDgywuuUv4Y5EEe7AAFlRaGqDZRPu\nSBPhDgyQFUFLBkkj3IEB8jy0kXBHwgh3YACO3JE6wh0YICsK1rgjacxeYIC8CFbKIGmEOzBAJyfc\nkTbCHRggL0JT3KgDCSPcgQGyIrSRW+whYcxeYIAsL1gtg6SNFO62d9k+bXvJ9sEBr3+P7b/qvf6Y\n7bm6CwUmiaWQSN3QcLc9JemQpN2SdkjaZ3vHmmF3S3ohIn5E0kckfajuQoFJyovgLkxI2vQIY3ZK\nWoqIM5Jk+4ikvZJO9Y3ZK+kDvZ8fkvRR246IqLFWSdLLnVwvd/K6fy3wCt+5kHPkjqSNEu6bJZ3t\ne7ws6a1lYyIis/2ipOskfaOOIvt98t+f1e89/OW6fy3wKjf/8BuaLgFYt1HCvTa290vaL0lbt25d\n1+/48Tdu0vt/Zm1XCKjfzm2EO9I1Srifk7Sl7/Fs77lBY5ZtT0u6RtLza39RRByWdFiS5ufn19Wy\nuXH2Gt04e816/lUAuGyMcsbohKTttrfZvkLSnZIW1oxZkPQrvZ9/XtK/jKPfDgAYzdAj914P/YCk\n45KmJH08Ik7afkDSYkQsSPpzSZ+yvSTpm+r+DwAA0JCReu4RcUzSsTXP3d/388uS3lFvaQCA9WIh\nLwC0EOEOAC1EuANACxHuANBChDsAtBDhDgAtRLgDQAsR7gDQQoQ7ALQQ4Q4ALUS4A0ALEe4A0EKE\nOwC0kJvadt32eUlfbeTNL80mjeH2gQm4HD83n/nykdLnvj4iZoYNaizcU2V7MSLmm65j0i7Hz81n\nvny08XPTlgGAFiLcAaCFCPfX7nDTBTTkcvzcfObLR+s+Nz13AGghjtwBoIUI90tg+17bYXtT07WM\nm+0HbX/Z9tO2/872tU3XNE62d9k+bXvJ9sGm6xk321tsP2L7lO2Ttu9puqZJsT1l+wnb/9h0LXUi\n3NfJ9hZJPyXpa03XMiFfkHRDRLxZ0lckva/hesbG9pSkQ5J2S9ohaZ/tHc1WNXaZpHsjYoekmyW9\n+zL4zCvukfRM00XUjXBfv49I+i1Jl8VJi4j4fERkvYePSpptsp4x2ylpKSLORMQFSUck7W24prGK\niK9HxJd6P/+vumG3udmqxs/2rKSflvRnTddSN8J9HWzvlXQuIp5qupaG/Jqkh5suYow2Szrb93hZ\nl0HQrbA9J+kmSY81W8lE/KG6B2lF04XUbbrpAr5b2f4nST8w4KX7JP22ui2ZVqn6zBHxD70x96n7\nJ/xnJlkbJsP2VZL+RtJ7IuLbTdczTrbfLum5iHjc9q1N11M3wr1ERNw+6HnbN0raJukp21K3PfEl\n2zsj4n8mWGLtyj7zCtvvlPR2SbdFu9fQnpO0pe/xbO+5VrO9Ud1g/0xE/G3T9UzALZL22L5D0pWS\nXm/70xFxV8N11YJ17pfI9rOS5iMilU2H1sX2LkkflvS2iDjfdD3jZHta3ZPGt6kb6ick/WJEnGy0\nsDFy90jlk5K+GRHvabqeSesduf9GRLy96VrqQs8do/qopKslfcH2k7Y/1nRB49I7cXxA0nF1Tywe\nbXOw99wi6Zck/UTvv++TvSNaJIojdwBoIY7cAaCFCHcAaCHCHQBaiHAHgBYi3AGghQh3AGghwh0A\nWohwB4AW+n9SUyHBkOuTZAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112331320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy  as np\n",
    "%matplotlib inline\n",
    "\n",
    "def step_function(x):\n",
    "    return np.array(x>0, dtype=np.int)\n",
    "\n",
    "x = np.arange(-5, 5, 0.1)\n",
    "y = step_function(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### シグモイド関数 (sigmoid function)\n",
    "\n",
    "$\n",
    "h (x) = \\frac{1}{1 + e^{-x}}\n",
    "$\n",
    "\n",
    "* NumPy配列に対応している\n",
    "    * ブロードキャストの機能により、スカラ値とNumPy配列の演算が行われるとスカラ値とNumPy配列の要素同士で演算が行われる\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHyFJREFUeJzt3Xd0XNXd7vHvz+qymm3JRZIr7hXbwgVCCRhimk2ohhU6\n2CRAgFBCe8kbyE2AJJRcuJQUigM4NiVxgsGUS3uplovci1wlN0mW1eto9v1DwlcY25LtkY5m5vms\nNcuaM0ea5yDpYWvPmX3MOYeIiISWTl4HEBGRwFO5i4iEIJW7iEgIUrmLiIQglbuISAhSuYuIhCCV\nu4hICFK5i4iEIJW7iEgIivTqiVNTU12/fv28enoRkaC0ePHiIudcWkv7eVbu/fr1Izs726unFxEJ\nSma2tTX7aVpGRCQEqdxFREKQyl1EJASp3EVEQpDKXUQkBKncRURCkMpdRCQEqdxFREKQyl1EJASp\n3EVEQpDKXUQkBKncRURCkMpdRCQEtVjuZvY3Mysws5UHedzM7E9mlmtmy81sXOBjiojI4WjNyP1F\nYOohHj8TGNR0mwk8c/SxRETkaLRY7s65T4HiQ+wyHXjZNfoKSDGzXoEKKCIihy8Qc+4ZQF6z+/lN\n277HzGaaWbaZZRcWFgbgqUVE5EDa9UpMzrnngecBsrKyXHs+t4hIINT5/JRW11NaXUdpdT1l1T7K\nauopq66nrMZHeY2Pitp6Kmp8VNQ2UFnro6rOR2VdA1W1PqrqG7j3rGFcnNW7TXMGoty3A81TZjZt\nExHp8JxzlFbXU1BeS0FZLQXlNRRV1FJUUUdRRS3FlXX7biVV9VTU+g759aIijMTYKDrHRNA5OpKE\nmEhS4qPJ6BJBfHQk8dER9E/t3ObHFYhynw/cZGZzgIlAqXNuZwC+rojIUatv8LOjpJptxVXk761m\n+95qtpdUs7O0ml2lNewsraHW5//e50VHdiK1czTdEmLo2jmaY9ISSImPokt8NCnxUSTH/f9bUlwU\nSbFRJMZGEhsV4cFRfl+L5W5mrwGnAKlmlg/8CogCcM49CywAzgJygSrg6rYKKyJyIM45dpbWkFtQ\nwcbCCjYXVe677Sipxt9sEjiik9EzKZZeybGMykzhjBGxdE+MoUdS479piTGkJsaQGBOJmXl3UEep\nxXJ3zl3awuMOuDFgiUREDqGi1seanWWs3lHG2l1lrN1Vzvpd5VTWNezbJzE2kv6pnRnXpwvnj82g\nd9f4fbceiTFERoT++zfb9QVVEZHDUefzs2pHKTl5JeTkl5KTX8Lmokpc00g8JT6KIT0SuXB8JgN7\nJDIwLYGB3RNITYgO6lF3IKjcRaTDqKz1kb11L19v2kP2lr3k5Jfsmw/vnhjDmN4pnHdsBiPSkxiR\nnkyPpJiwL/GDUbmLiGca/I5leSV8ur6Q/8ktIievBJ/fEdnJGJGRzOWT+jK+bxfG9ulCz+RYr+MG\nFZW7iLSr0up6Pl5XwAdrCvh0fSGl1fV0MhiVmcL1Jw1g8oBuZPXrQny06ulo6L+eiLS54so6Fq7a\nxYIVO/ly4x58fkdqQjRnDO/ByUPS+MHAVFLio72OGVJU7iLSJqrqfCxctYu3lu7g89wiGvyOft3i\nue7EAZw+vAdje6fQqZPmy9uKyl1EAsY5x+Kte3ntmzzeWbmTqroGMrvEMeukAZw9uhfDeyXpBdB2\nonIXkaNWVlPP69n5vPbNNjYUVNA5OoJzR6dzwfhMsvp20QjdAyp3ETlim4sqefHzzby+OJ/KugbG\n9E7hkQtGcc7odDrHqF68pP/6InLYcvJKePaTjby7aheRnYxzR6dz1Qn9GJ2Z4nU0aaJyF5FWW7Sl\nmCc+WM/nuXtIio3kxlMGcsXxfemeqHPQOxqVu4i0aOm2vTz2/no+21BEakIM95w5lMsm9iExNsrr\naHIQKncROajNRZU8+u5a3lm5i66do7n3rKFcPqkfcdEdY1lbOTiVu4h8T0lVHU98sIG/f7WV6MhO\n3DZlMNed2F8vkgYRfadEZB+/3/GP7DwefXctpdX1zJjQh1unDNKcehBSuYsIACu3l3LfWyvIyS9l\nQr+u/Hr6CIb1SvI6lhwhlbtImKupb+CJDzbw58820SU+micuOZbpx6brnaRBTuUuEsaytxRz5+vL\n2VxUycVZmdx31nCS43UGTChQuYuEoTqfn8c/WM9zn2wko0scr1w3kRMGpnodSwJI5S4SZjbsLufn\nc5axZmcZM47rzf3nDCdBZ8GEHH1HRcKEc455i/N54F8r6RwdyZ+vyOL04T28jiVtROUuEgYqa338\n1z9X8ubS7Uwe0I0nZxxL9ySd3hjKVO4iIW5zUSWzZmeTW1DBbVMGc9OpA4nQErwhT+UuEsI+WlvA\nz+csJbKT8fI1E/nBIL1oGi5U7iIhyDnHM59s5PcL1zGsZxLPXT6e3l3jvY4l7UjlLhJi6nx+7n1r\nBa8vzmfamHQeuWC0FvoKQyp3kRBSUlXHrNmL+XpzMbdOGcQtpw3SO03DVKfW7GRmU81snZnlmtnd\nB3i8j5l9ZGZLzWy5mZ0V+KgicijbS6q54JkvWLqthCcuOZZbpwxWsYexFkfuZhYBPA2cDuQDi8xs\nvnNudbPd7gfmOueeMbPhwAKgXxvkFZEDWL+7nCv/9g0VtT5evnYCkwZ08zqSeKw1I/cJQK5zbpNz\nrg6YA0zfbx8HfLt8XDKwI3ARReRQFm8t5qJnv6TB75g7a7KKXYDWzblnAHnN7ucDE/fb57+B98zs\nZqAzMCUg6UTkkL7ILeLal7LpmRzLy9dM0Bkxsk+r5txb4VLgRedcJnAWMNvMvve1zWymmWWbWXZh\nYWGAnlokPH20toCrXlxEn67xzJ01WcUu39Gact8O9G52P7NpW3PXAnMBnHNfArHA994t4Zx73jmX\n5ZzLSktLO7LEIsK7K3cxc3Y2g3sk8NrMSaQlxngdSTqY1pT7ImCQmfU3s2hgBjB/v322AacBmNkw\nGstdQ3ORNvDeql3c9OoSRqQn88p1k+jaOdrrSNIBtVjuzjkfcBOwEFhD41kxq8zsQTOb1rTb7cD1\nZpYDvAZc5ZxzbRVaJFx9tLaAG19dwoiMZF6+dgLJcbqwhhxYq97E5JxbQOPpjc23PdDs49XACYGN\nJiLNfbq+kFl/X8yQnom8fM0EkmJV7HJwgXpBVUTa0KItxcycnc0xaQn8/dqJGrFLi1TuIh3cqh2l\nXPPiItKT45h97QRS4jXHLi1TuYt0YJuLKrnyb9+QEBPJ7Osmkpqgs2KkdVTuIh1UQVkNl//1a/wO\nZl87kYyUOK8jSRBRuYt0QBW1Pq5+cRHFlXW8ePVxDOye4HUkCTJa8lekg6lv8POzV5awdlc5f7ki\ni9GZKV5HkiCkkbtIB+Kc4943V/Dp+kL+13kj+eHQ7l5HkiClchfpQJ75ZCPzFufz81MHMmNCH6/j\nSBBTuYt0EO+s2Mmj765j2ph0bjt9sNdxJMip3EU6gOX5Jdw2dxnj+qTw6IWjdQUlOWoqdxGP7S6r\n4bqXsunWOYbnLs8iNkoXs5ajp7NlRDxUU9/ArNmLqaj18ebPjtfSvRIwKncRjzjn+K9/rmRZXgnP\n/mQcQ3smtfxJIq2kaRkRj7z0xZZ9Z8ZMHdnL6zgSYlTuIh74etMeHnp7DVOG9eDWKTozRgJP5S7S\nznaX1XDjq0vp2zWexy8ZQ6dOOjNGAk9z7iLtqL7Bz42vLKGy1ser108kURfckDaichdpR79dsIbs\nrXv506VjGdwj0es4EsI0LSPSTt5evpMXPt/C1Sf0Y9qYdK/jSIhTuYu0g81FlfzyjeWM7ZPCPWcO\n8zqOhAGVu0gbq6lv4MZXlhAZYTx12TiiI/VrJ21Pc+4ibezB/6xm9c4y/npllq6mJO1GQwiRNvTv\nnB28+vU2Zp00gNOG9fA6joQRlbtIG8krruLeN1cwtk8Kd/xoiNdxJMyo3EXaQH2Dn5tfWwoGf5ox\nlqgI/apJ+9Kcu0gb+ON761mWV8LTl42jd9d4r+NIGNJwQiTAPttQyLOfbOTSCX04e7QWBBNvtKrc\nzWyqma0zs1wzu/sg+1xsZqvNbJWZvRrYmCLBobiyjtvn5jCwewIPnDPc6zgSxlqcljGzCOBp4HQg\nH1hkZvOdc6ub7TMIuAc4wTm318x0yXYJO8457np9OSVV9bx49QTionVFJfFOa0buE4Bc59wm51wd\nMAeYvt8+1wNPO+f2AjjnCgIbU6Tje+XrbXywZjd3TR3C8HRdeEO81ZpyzwDymt3Pb9rW3GBgsJl9\nbmZfmdnUQAUUCQa5BRX85u3VnDgolWtO6O91HJGAnS0TCQwCTgEygU/NbJRzrqT5TmY2E5gJ0KdP\nnwA9tYi36nx+bv3HUuKiIvjjRVqfXTqG1ozctwO9m93PbNrWXD4w3zlX75zbDKynsey/wzn3vHMu\nyzmXlZaWdqSZRTqUJz9cz8rtZTx8wWi6J8V6HUcEaF25LwIGmVl/M4sGZgDz99vnnzSO2jGzVBqn\naTYFMKdIh5S9pZhnPt7IxVmZ/GhET6/jiOzTYrk753zATcBCYA0w1zm3ysweNLNpTbstBPaY2Wrg\nI+BO59yetgot0hGU19Rz29xlZHaJ54FzR3gdR+Q7WjXn7pxbACzYb9sDzT52wC+abiJh4aH/rGb7\n3mrm3TCZhBi92Vs6Fr1DVeQIvLdqF3Oz8/npKccwvm9Xr+OIfI/KXeQwFVXUcs+bKxjeK4lbThvs\ndRyRA9LfkiKHwTnHvW+uoLzGx6vXH6urKkmHpZ9MkcPwxpLtvLd6N3f+aAhDeiZ6HUfkoFTuIq20\nvaSaX89fxYT+XbnmB3oXqnRsKneRVvD7HXfOy6HBOf540Rgi9C5U6eBU7iKtMPurrXyxcQ/3nz1c\nF9+QoKByF2nBpsIKfvfOGk4ZksalE3q3/AkiHYDKXeQQGvyO2+flEBMZwSMXjMZM0zESHHQqpMgh\nPP/pJpZuK+HJGcfSQ4uCSRDRyF3kINbuKuPx99dz1qieTBuT7nUckcOichc5gDqfn9vn5pAUF8lD\n00dqOkaCjqZlRA7gqY9yWbWjjOcuH0+3hBiv44gcNo3cRfazPL+Epz/K5fyxGVqjXYKWyl2kmZr6\nBn4xN4e0hBh+pTXaJYhpWkakmcfeX09uQQUvXTOB5Pgor+OIHDGN3EWaLNpSzJ8/28RlE/tw8mBd\n41eCm8pdBKis9XHHvBwyu8Rx71nDvI4jctQ0LSMCPPzOWrYVV/Ha9ZN0yTwJCRq5S9j7bEMhs7/a\nyrUn9GfSgG5exxEJCJW7hLXS6nrunLecgd0TuONHQ7yOIxIwKncJa7/+9yoKK2p57OIxxEZFeB1H\nJGBU7hK23l25kzeXbOfGHw5kdGaK13FEAkrlLmGpoLyGe99ayaiMZG4+daDXcUQCTuUuYcc5xz1v\nrKCi1sfjl4whKkK/BhJ69FMtYecfi/L4cG0Bv5w6lIHdE72OI9ImVO4SVrbtqeKh/6xm8oBuXH18\nP6/jiLQZlbuEDV+Dn9vmLqNTJ+MPF4+hUyet0S6hq1XlbmZTzWydmeWa2d2H2O8CM3NmlhW4iCKB\n8ewnG1m8dS+/OW8kGSlxXscRaVMtlruZRQBPA2cCw4FLzWz4AfZLBG4Bvg50SJGjtTy/hCc+2MC5\nY9KZfmyG13FE2lxrRu4TgFzn3CbnXB0wB5h+gP0eAh4BagKYT+SoVdc1cOs/lpGWGMNvpo/0Oo5I\nu2hNuWcAec3u5zdt28fMxgG9nXNvH+oLmdlMM8s2s+zCwsLDDityJB56ezWbiyr5w0VjtEa7hI2j\nfkHVzDoBjwG3t7Svc+5551yWcy4rLU3rZUvbW7hqF69+vY2ZJw7ghIGpXscRaTetKfftQO9m9zOb\ntn0rERgJfGxmW4BJwHy9qCpe211Ww91vLGdkRhK3n6FFwSS8tKbcFwGDzKy/mUUDM4D53z7onCt1\nzqU65/o55/oBXwHTnHPZbZJYpBX8fscd83Korm/gyRljiY7UWb8SXlr8iXfO+YCbgIXAGmCuc26V\nmT1oZtPaOqDIkXj+s018tqGIB84ZwTFpCV7HEWl3rbrkjHNuAbBgv20PHGTfU44+lsiRW7ptL39Y\nuI6zRvXk0gm9W/4EkRCkv1UlpJTV1PPzOUvpkRTL784fjZnehSrhSReLlJDhnOO+t1ayo6SGubMm\nkxyn0x4lfGnkLiFjzqI8/p2zg1+cPpjxfbt4HUfEUyp3CQmrd5Txq/mrOHFQKj89+Riv44h4TuUu\nQa+8pp4bX11Cl/gonrjkWK32KILm3CXIOee4+80VbCuu4rXrJ9EtIcbrSCIdgkbuEtRe/nIrby/f\nye1nDGZC/65exxHpMFTuErQWby3mof+s5rSh3bnhJM2zizSncpegVFhey89eWUJ6ShyPaZ5d5Hs0\n5y5Bx9fg5+bXllBSVc+bPztO57OLHIDKXYLOw++s5atNxfzhojGMSE/2Oo5Ih6RpGQkqby7J5y//\ns5krJ/flwvGZXscR6bBU7hI0lueXcPebK5g0oCv3n/O9y/iKSDMqdwkKheW1zJq9mLSEGJ6+bBxR\nEfrRFTkUzblLh1dT38DM2dnsrarj9RuO1xuVRFpB5S4dmnOOu15fztJtJTz7k3GMzNALqCKtob9t\npUN78sMNzM/ZwV1ThzB1ZC+v44gEDZW7dFj/WradJz7YwAXjMrXSo8hhUrlLh/TFxiLumJfDhP5d\n+e35I3VFJZHDpHKXDmftrjJmvbyYft068+fLs4iJjPA6kkjQUblLh7KztJqrX1hEXHQEL14zgeR4\nLS0gciR0tox0GHsr67jir99QXuPjH7MmkZES53UkkaClcpcOoaLWx1UvLmJrcRUvXn2c1owROUqa\nlhHP1foamDU7m5XbS3nq0rEcf0yq15FEgp7KXTxV3+Dn5leX8nnuHh69YDRnjOjpdSSRkKByF8/4\nGvzcMmcp763eza+njeACrfIoEjAqd/GEr8HPbXNzWLBiF/efPYwrj+/ndSSRkNKqcjezqWa2zsxy\nzezuAzz+CzNbbWbLzexDM+sb+KgSKnwNfm6fl8O/c3Zw95lDue7EAV5HEgk5LZa7mUUATwNnAsOB\nS81s/8W0lwJZzrnRwOvAo4EOKqGhzufn5teW8q9lO7jzR0O4QcsKiLSJ1ozcJwC5zrlNzrk6YA4w\nvfkOzrmPnHNVTXe/AjR5Kt9TU9/AT/++mHdWNk7F3PjDgV5HEglZrSn3DCCv2f38pm0Hcy3wzoEe\nMLOZZpZtZtmFhYWtTylBr6LWx7UvLeLDtQX85ryRmooRaWMBfROTmf0EyAJOPtDjzrnngecBsrKy\nXCCfWzquooparn5hEat3lvHHi8borBiRdtCact8O9G52P7Np23eY2RTgPuBk51xtYOJJsMsrruLy\nv37NrrIa/nzFeE4d2sPrSCJhoTXlvggYZGb9aSz1GcBlzXcws7HAc8BU51xBwFNKUFqWV8J1L2Xj\n8/t55bpJjO/bxetIImGjxTl355wPuAlYCKwB5jrnVpnZg2Y2rWm33wMJwDwzW2Zm89sssQSFt5fv\n5JLnviQ+OoLXb5isYhdpZ62ac3fOLQAW7LftgWYfTwlwLglSzjn+z8cb+f3CdYzv24XnLx+vC1qL\neECrQkrAVNb6uOv15by9YifTxqTz6IWjiY3ShTZEvKByl4DYUlTJzNnZ5BZUcM+ZQ5l50gBdGk/E\nQyp3OWrvrtzJna8vJ6KT8fI1E/nBIC3ZK+I1lbscsZr6Bn63YA0vfbmVMZnJPHXZOHp3jfc6loig\ncpcjtGF3ObfMWcbqnWVc94P+3DV1KNGRWmRUpKNQucth8fsdL3yxhUfeXUtCTCR/uSKLKcP1xiSR\njkblLq2WV1zFL99Yzhcb9zBlWHd+d/5o0hJ1mqNIR6RylxY1+B0vfL6ZP763nk4GD58/ikuO662z\nYUQ6MJW7HNKK/FLu/+cKcvJLOXVod35z3kjSU+K8jiUiLVC5ywGVVNXx+4XrePWbbXTrHM2fLh3L\nuaN7abQuEiRU7vIddT4/r369lSc/3EBZjY+rju/HbacPJik2yutoInIYVO4CNK4J8+7KXTzy7lq2\n7Kli8oBu/GracIb2TPI6mogcAZV7mHPO8fH6Qh5/fz3L80sZ1D2BF646jlOGpGkKRiSIqdzD1Lel\n/r8/3MCSbSVkdonj0QtGc/64DCIj9GYkkWCncg8zvgY/b6/YyTMfb2TtrnLSk2P57Y9HceH4TL3D\nVCSEqNzDxN7KOuYsymP2l1vYUVrDwO4J/OGiMUwbk65SFwlBKvcQ5pxjybYS5nyzjX8v30FNvZ/j\nj+nGr6eP5LSh3enUSXPqIqFK5R6CCsprmL9sB/Oy81m3u5z46Ah+PDaTq47vx5CeiV7HE5F2oHIP\nEeU19Xy4poB/LtvOZxuKaPA7xmQm87vzR3HumHQSYvStFgkn+o0PYnsr6/hoXQELVuzi0w2F1Pn8\npCfHcsPJA/jx2AwGdtcoXSRcqdyDiHOOdbvL+WRdIR+uLSB7SzF+Bz2TYvnJxL6cPbonY3t30Vy6\niKjcO7qdpdV8uXEPX2zcw2cbCtldVgvA0J6J3PjDgUwZ1oNRGckqdBH5DpV7B+L3OzYVVZC9ZS+L\ntuwle2sxW/dUAZASH8UJx6Ry0uBUThqcRq9krcwoIgencveIc45txVWs2lHGyu2l5OSXsDyvlPJa\nHwBdO0czvm8XLp/Ul8nHdGNYzySNzkWk1VTubcw5R1FFHbkFFeQWlLN2Vznrmm7fFnlkJ2Nor0Sm\nHZvOmN4pjO/bhQGpnbW2i4gcMZV7ADjn2FNZR15xFduKq9i6p4otRZVs3lPJ5qJKSqrq9+2bGBvJ\n0J6JTB+bzoj0ZEamJzOoRwKxUREeHoGIhBqVewv8fsfeqjp2l9VSUF7D7rIadpbWsKu0hh2lNWzf\nW8WOkhqq6xu+83npybH0S+3MWaN6MTAtgYHdG2+9kmM1IheRNteqcjezqcCTQATwF+fcw/s9HgO8\nDIwH9gCXOOe2BDbq0fP7HZV1Pkqr6xtvVfWUVNezt6qOkqp69lTUUVxZy57KOvZU1FFUUUtxZR0+\nv/vO1zGD1IQYeiXHMrhHIqcM6U5GShx9u8XTp2s8mV3iiYvWSFxEvNNiuZtZBPA0cDqQDywys/nO\nudXNdrsW2OucG2hmM4BHgEvaInBecRUbCsqpqmugqq6B6n3/+qisa6Cy1kdFrW/fv+U1jf+WVddT\nUetjv57+jvjoCLp2jqZb52h6JccyKiOZ1MRo0hJi6J4US4+kGLonxtIjKVaLbYlIh9aakfsEINc5\ntwnAzOYA04Hm5T4d+O+mj18HnjIzc84dokqPzNsrdvLwO2u/t90M4qMi6BwTSUJMJPExESTGRNG7\nazyJMZEkxUWRGBtJYmwkKXHRJMVFkRwXRUp8FF3io0mJj9K8t4iEjNaUewaQ1+x+PjDxYPs453xm\nVgp0A4oCEbK5847NYPKAbsRFRxAXFUFcdASdoyOJjeqkuWwRkSbt+oKqmc0EZgL06dPniL5Gz+RY\neibHBjKWiEjIac3E8Xagd7P7mU3bDriPmUUCyTS+sPodzrnnnXNZzrmstLS0I0ssIiItak25LwIG\nmVl/M4sGZgDz99tnPnBl08cXAv+3LebbRUSkdVqclmmaQ78JWEjjqZB/c86tMrMHgWzn3Hzgr8Bs\nM8sFimn8H4CIiHikVXPuzrkFwIL9tj3Q7OMa4KLARhMRkSOlk7VFREKQyl1EJASp3EVEQpDKXUQk\nBKncRURCkMpdRCQEqdxFREKQyl1EJASp3EVEQpDKXUQkBKncRURCkMpdRCQEqdxFREKQebXsupkV\nAls9efKjk0obXD4wCITjceuYw0cwHXdf51yLVzvyrNyDlZllO+eyvM7R3sLxuHXM4SMUj1vTMiIi\nIUjlLiISglTuh+95rwN4JByPW8ccPkLuuDXnLiISgjRyFxEJQSr3o2Bmt5uZM7NUr7O0NTP7vZmt\nNbPlZvaWmaV4naktmdlUM1tnZrlmdrfXedqamfU2s4/MbLWZrTKzW7zO1F7MLMLMlprZf7zOEkgq\n9yNkZr2BM4BtXmdpJ+8DI51zo4H1wD0e52kzZhYBPA2cCQwHLjWz4d6manM+4Hbn3HBgEnBjGBzz\nt24B1ngdItBU7kfuceAuICxetHDOveec8zXd/QrI9DJPG5sA5DrnNjnn6oA5wHSPM7Up59xO59yS\npo/LaSy7DG9TtT0zywTOBv7idZZAU7kfATObDmx3zuV4ncUj1wDveB2iDWUAec3u5xMGRfctM+sH\njAW+9jZJu3iCxkGa3+sggRbpdYCOysw+AHoe4KH7gHtpnJIJKYc6Zufcv5r2uY/GP+Ffac9s0j7M\nLAF4A7jVOVfmdZ62ZGbnAAXOucVmdorXeQJN5X4QzrkpB9puZqOA/kCOmUHj9MQSM5vgnNvVjhED\n7mDH/C0zuwo4BzjNhfY5tNuB3s3uZzZtC2lmFkVjsb/inHvT6zzt4ARgmpmdBcQCSWb2d+fcTzzO\nFRA6z/0omdkWIMs5FyyLDh0RM5sKPAac7Jwr9DpPWzKzSBpfND6NxlJfBFzmnFvlabA2ZI0jlZeA\nYufcrV7naW9NI/c7nHPneJ0lUDTnLq31FJAIvG9my8zsWa8DtZWmF45vAhbS+MLi3FAu9iYnAJcD\npzZ9f5c1jWglSGnkLiISgjRyFxEJQSp3EZEQpHIXEQlBKncRkRCkchcRCUEqdxGREKRyFxEJQSp3\nEZEQ9P8A9nnclg8mQC4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112331828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy  as np\n",
    "%matplotlib inline\n",
    "\n",
    "def sigmoid_function(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "x = np.arange(-5, 5, 0.1)\n",
    "y = sigmoid_function(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### シグモイド関数とステップ関数の比較\n",
    "\n",
    "* 滑らかさ\n",
    "    * ステップ関数 (パーセプトロン)\n",
    "        * 0を堺に急激に出力が変わっている\n",
    "            * 出力は0か1のみ\n",
    "    * シグモイド関数 (ニューラルネットワーク)\n",
    "        * 滑らかな曲線\n",
    "        * 入力に対して連続的に出力が変化している\n",
    "            * 0.3245など実数値を出力する\n",
    "* 共通点\n",
    "    * 入力信号が重要な値であれば大きな値を出力し、重要でなければ小さな値を出力する\n",
    "    * どんなに入力値が大きくても小さくても、出力信号の値は0〜1\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH9ZJREFUeJzt3Xl8VPW5x/HPkxD2VYiCBAwuaHFXxH2pS0VrxbWCxaq1\npYva2trbq7VXrfa29va2WluvLbXVVq1UcYtIxbpX68KiooDYgChBFGSHAJmZ89w/TgIBycyQzOTk\nnHzfr9e8kpk5mXnmZfL1x3N+5/czd0dERJKlJOoCRESk8BTuIiIJpHAXEUkghbuISAIp3EVEEkjh\nLiKSQAp3EZEEUriLiCSQwl1EJIE6RPXG/fr188rKyqjeXkQklqZPn/6Ju5fnOi6ycK+srGTatGlR\nvb2ISCyZ2fv5HKe2jIhIAincRUQSSOEuIpJACncRkQRSuIuIJJDCXUQkgRTuIiIJpHAXEUkghbuI\nSAIp3EVEEkjhLiKSQAp3EZEEUriLiCRQznA3sz+Z2RIze7uJ583MbjWzajObaWYHFb5MERHZHvmM\n3O8CRmZ5/hRgj/rbOOD2lpclIiItkXM9d3d/wcwqsxwyCviLuzvwipn1NrMB7r64QDWKRGpVbYqZ\ni1ZGXYYkyG7l3dm5d5eivkchNusYCCxsdL+m/rFPhbuZjSMc3TN48OACvLVI8d34+GwmTq+JugxJ\nkJ+csQ9jD9ulqO/Rqjsxuft4YDzA8OHDvTXfW6S5Vq9PsUvfrvzy3P2jLkXaAMvUUVq3ig4bV4Vf\n61ZTuum2Jryl1lKaWkNJah2lqXWUptdRkqqlJF1LabqWWr8euLiodRYi3BcBgxrdr6h/TCQRMoHT\no3MHhlfuEHUpUgzusH4FrP0Y1nwUfl27BNYtgXWfhLfaT6B2GdSugLo12V+vpAw694SO3aFTj/Br\n9x2hY7fw+7Ku9Nx5r6J/rEKEexVwmZlNAA4FVqnfLkmSCpzSEs0ajq1MClYthBULYOUHsHJheH/1\nh7B6Ufg1veHTP1faCbqVQ7d+4a3fUOiyA3TdAbr0CW+de0OX3tC5V3jr1BPKOrf6R9yWnOFuZvcB\nxwH9zKwGuA4oA3D33wGTgVOBaqCWYv9bQ6SVZYKAshKLugzJxj0M6qVz4ZN/w7Lq8LZ8HqyqAQ82\nH2ul0HNn6DkQdj4Q9vo89BgAPfpD9/7QfSfoXh4GtcX3v3s+s2XG5HjegUsLVpFIG5PKOKUK97Zj\n4xr46G346C34+G1YMhuWzIG6tZuP6dQL+u4KFSNgv9HQpxL67AK9dwmDvLRVTzdGIvmfUKSFMoHT\nuUxtmUik6+CjmbBoev1tRjgip34+Rpc+sOPecMD5UL4n9Nsz/NqtPNaj7kJQuIvkkA6cDuq5t46N\na2HhK7DgJfjgFfhwxuZ+ePf+MPBg2O+L0H8/GLBfOApv5yHeFIW7SA7pTEAHtWWKI8iEI/Lqp2H+\ns+H3QRpKOsCA/eGQr8KgEVBxSNgnl7wp3EVyyATquRfU+pVQ/RTMnRyG+oaVYCXhyc0jLofKo2Hw\nYeHUQWk2hbtIDunAKStVW6ZF1i2Ddx6DWY/Agn+Go/Nu5bDXabD7CbDrceEUQykYhbtIDulMoJF7\nc9StgzmTYObfYP5z4BnYYVc4/LJw+uHA4aBzGUWjcBfJITyhqnDPizssfBWm/xlmPwqpddB7MBz5\nHdj7TOi/r06AthKFu0gOmcDpUKpAymrDKnjjrzD9Llj6TniZ/T5nhVMUBx2mEXoEFO4iOYQXMSmc\ntmnZPHj1d2Gw160Npyqe/hvY+yzo1D3q6to1hbtIDplAUyE/ZdF0ePEWmPNYOG1xn7Ph0K/DQG3E\n1lYo3EVySKsts9n7L8NzP4P3ng8Xyjr6ShgxDnrsFHVlshWFu0gO6YxOqFIzDZ79b5j3DHTbEU66\nAQ6+OFzaVtokhbtIDpn2vOTvsnnw1PUwpwq69oWTbgyvGu3YNerKJAeFu0gO6SCgrL21ZWqXw/M/\nh6l3hOuaH/dDOPxSnSSNEYW7SBZB4ARO+7mIKQjg9b/AUz8OlwU46EI47mr11GNI4S6SRToIl5Zt\nFz33xW/CY1eEKzEOPgJO/QX03yfqqqSZFO4iWWQawj3Ja8uk1sNzN8G/fhP21c/6A+x7rq4kjTmF\nu0gWqSDcni2xI/cPXoFHvhVuR3fgWPjcT8INMCT2FO4iWWQy4cg9cT33dB0891N46dfQaxB8+dFw\nZUZJDIW7SBbpJLZllrwDD34VPn4LDvoynPxT6NQj6qqkwBTuIlmkk9SWcYc37oXHvx9uhDH6Ptjr\n1KirkiJRuItkkU5KW2bjWnj8Spg5Idzp6Ow7oEf/qKuSIlK4i2TRMFsm1hcxLZsHE74En8wNL0Y6\n5vtQUhp1VVJkCneRLBraMrFdfuDdJ8P+ekkpjH0Idvts1BVJK1G4i2QR24uY3OHFm+HpG8ILkc67\nF/rsEnVV0ooU7iJZNPTcYxXu6TqYdEV48nSfc8LNM7TQV7ujcBfJYvNUyJiEe+1y+NsF8P6L4Zow\nx/6nrjRtp/JqJJrZSDOba2bVZnbVNp4fbGbPmtnrZjbTzDS/ShIhs2kqZAx67isXwp9OhprXwiUE\njrtKwd6O5Ry5m1kpcBtwElADTDWzKnef3eiwHwH3u/vtZjYMmAxUFqFekVaViktbZskcuOfscMrj\nBQ9D5VFRVyQRy2c4MgKodvf57l4HTABGbXWMAw1bsvQCPixciSLRaZgK2abnuX/wKvxpJAQZuHiy\ngl2A/HruA4GFje7XAIdudcz1wJNmdjnQDTixINWJRKzNLz8w/3m4bzT0GBCO2DUjRuoV6jd2DHCX\nu1cApwJ3m9mnXtvMxpnZNDObtnTp0gK9tUjxpDNtePmBd5+Ee8+FPpXwlScU7LKFfMJ9ETCo0f2K\n+scauwS4H8DdXwY6A/22fiF3H+/uw919eHl5efMqFmlF6bbalpnzGEw4H3bcCy6cBN13jLoiaWPy\nCfepwB5mNsTMOgKjgaqtjvkAOAHAzD5DGO4amkvsbV5+oA21Zd55HB64CAbsD1+ugm59o65I2qCc\nv7HungYuA6YAcwhnxcwysxvM7PT6w64EvmZmbwL3ARe5uxeraJHWkso0LD/QRkbu7z4J918YBvsF\nD0GX3lFXJG1UXhcxuftkwumNjR+7ttH3s4EjC1uaSPQybWn5geqn4W9jYadh4ToxnXtFXZG0YW3o\n35oibU+buUL1/ZfDlR37DYULHtGIXXJSuItksXltmQj/VBbPhL+eB70GhtMdu+4QXS0SGwp3kSwy\nQcQ992Xz4J6zoFP3cMTeXbPMJD8Kd5Es0lFu1rHmI7j7DPAgDPbeg3L/jEg9rQopkkVk2+xtXBNe\noLRuGVw0CcqHtu77S+wp3EWy2LxZRyv+IzeTCqc7fjwLxkyAgQe13ntLYijcRbLYtORva7Vl3OGx\nK2De0/CFW2Ho51rnfSVx1HMXyaJhyd/S1loX/cWb4Y174JgfwMEXts57SiIp3EWyyAROiUFJa/Tc\nZz8KT/843Brvsz8s/vtJoincRbJIB946y/0umgEPfR0qRsCo27SDkrSYwl0ki3QmKP7SA6sXw31j\noFs5jL4XyjoX9/2kXdAJVZEs0oEXdxpkakO4XszGNfDVf2jpXikYhbtIFpnAi7fcrzs8fiUsmgZf\nvBt22rs47yPtktoyIlmkg6B4I/fXxm+eGTPs9NzHi2wHhbtIFumMF6fnvuAleOJq2PNUOO7qwr++\ntHsKd5EsMoEX/gKm1YvDnZR2GAJn/h6iXHFSEks9d5EsUoEXdumBTCoM9rq1cGEVdO5ZuNcWaUTh\nLpJFptA99yf/Cxa+Amf/EXb8TOFeV2Qr+vegSBYF7bnPehhevR0O/Sbse05hXlOkCQp3kSzSheq5\nL5sHj14OFYfASTe0/PVEclC4i2SRLkTPPbUBHrgQSjvAOXdCh46FKU4kC/XcRbIoyPIDT1wFH70F\nY/6m3ZSk1WjkLpJFi5cfePtBmH4nHPFt2HNk4QoTyUHhLpJFi5YfWLEg3Hij4hA44dqC1iWSi8Jd\nJIt0pplTITMpmHgJYOG0x9Kygtcmko167iJZhCdUmxHuz/wkXBDs3Lugzy4Fr0skF43cRbJo1vID\n856Bl26Bgy+Cvc8sSl0iueQV7mY20szmmlm1mV3VxDFfNLPZZjbLzP5a2DJFopHKBNs3FXLdMnj4\nm9BvTzj5Z8UrTCSHnG0ZMysFbgNOAmqAqWZW5e6zGx2zB3A1cKS7rzAz7TggiZDZntky7lB1Gaxf\nDmMnQseuxS1OJIt8hiQjgGp3n+/udcAEYNRWx3wNuM3dVwC4+5LClikSje26QnXan2DuZDjxeui/\nbzHLEskpn3AfCCxsdL+m/rHGhgJDzewlM3vFzDShVxIh77Vllr4LU66B3Y4P144RiVihZst0APYA\njgMqgBfMbF93X9n4IDMbB4wDGDx4cIHeWqR4wouYcoyB0nXw0FehrAuccbvWZ5c2IZ/fwkVA42um\nK+ofa6wGqHL3lLu/B7xLGPZbcPfx7j7c3YeXl5c3t2aRVpMJAspytWWevwkWvwmn/wZ69G+dwkRy\nyCfcpwJ7mNkQM+sIjAaqtjrmEcJRO2bWj7BNM7+AdYpEIp3JcUL1g1fgxZvhwLHwmdNarzCRHHKG\nu7ungcuAKcAc4H53n2VmN5hZw66+U4BlZjYbeBb4D3dfVqyiRVpL1ouYNqyGh8ZB78Ew8qbWLUwk\nh7x67u4+GZi81WPXNvrege/V30QSI7yIqYkx0JSrYdVCuPgJ6NSjdQsTyUFnfkSySAVNLPn7zuPw\n+j1w1Hdh8KGtX5hIDgp3kSYEgePOp3vua5dC1bfDuezHbvOCbZHIaeEwkSakAwfYcslfd5h0BWxc\nDWc+pl2VpM3SyF2kCekgALYaub95H7wzKVyffadhEVUmkpvCXaQJDSP3TT33lQvh7/8JuxwJh30r\nwspEclO4izQhk2kU7kEAj34Lggyc8X9QUhpxdSLZqecu0oRUQ1umtASm3gHvvQCn3QJ9KqMtTCQP\nCneRJmTq2zJ9at+Hf10Lu58UbsAhEgNqy4g0IZ1xSgg4fOaPoEOncO0Ya8aWeyIR0MhdpAnpwBlX\nOom+K98MN7nuOSDqkkTyppG7SBNKl87iux0m8uHOJ8M+Z0ddjsh2UbiLbEu6jh2f/i6r6cqcg65T\nO0ZiR+Eusi3//F86f/I216QuIdOlb9TViGw3hbvI1hbNgBf+l+W7n8WTwSH576Eq0oYo3EUaS22A\nh78B3XfigxHhqtYdtG2exJBmy4g09uxP4JO5MPZBNpT0BMhvg2yRNkZDEpEG778M//otHHwx7H7i\npouYmtysQ6QN02+tCMDGtfDIN8Mt8z53IwCpzDZWhRSJCbVlRACeug5WLICLJm3aMi+z9aqQIjGi\nkbvIvGfChcEOvxQqj9r08KYlfzVbRmJI4S7t2/qV8Mil0G9POP5HWzyV3rTkr/5MJH7UlpH27Ymr\nYO3HMPpeKOuyxVPb3IlJJCY0JJH2a3ZVuG3eMd+HgQd96unMpj1UFe4SPwp3aZ/WfBxudD3gADjm\nP7Z5SENbRiN3iSOFu7Q/7vDYt8Ppj2eNh9KybR62eQ9V/ZlI/KjnLu3PjL/Au0/AyT+D8j2bPCxT\n33PXbBmJIw1JpH1Z/h5M+SFUHg2HfiProamM5rlLfCncpf3IpOHhr4OVwhm3Q452S8MJVfXcJY7y\nCnczG2lmc82s2syuynLc2WbmZja8cCWKFMhLN8PCV+Hzv4Teg3Ient40W0ZjIImfnL+1ZlYK3Aac\nAgwDxpjZsG0c1wP4DvBqoYsUabFFM+C5m8Lt8vY7N68fSWttGYmxfIYkI4Bqd5/v7nXABGDUNo67\nEfg5sKGA9Ym0XF0tPDQOuu8UjtrzlNbaMhJj+YT7QGBho/s19Y9tYmYHAYPc/fFsL2Rm48xsmplN\nW7p06XYXK9IsU34Iy6rhjP+DLn3y/rFM4JSWGKb9UyWGWtxMNLMS4FfAlbmOdffx7j7c3YeXl5e3\n9K1FcpszCabfCUdcDrset10/mgoCtWQktvIJ90VA47NPFfWPNegB7AM8Z2YLgMOAKp1UlcitXgxV\nl8OA/eH4/9ruH89kXC0Zia18wn0qsIeZDTGzjsBooKrhSXdf5e793L3S3SuBV4DT3X1aUSoWyUcQ\nhJtvpNbD2X+EDh23+yXSgcJd4itnuLt7GrgMmALMAe5391lmdoOZnV7sAkWa5V+3wvxnYeTPoN8e\nzXqJdBBoiz2JrbyWH3D3ycDkrR67toljj2t5WSItUDMNnrkRho2Cgy9q9ss0nFAViSMNSyRZNqyC\niV+BHjvDF26FFsx0SWecMoW7xJQWDpPkcIdJ34VVNfCVJ6BL7xa9XDpwSrVomMSURu6SHDP+DG8/\nCMdfA4NGtPjlwhOq+hOReNJvriTDR2/B5B/AbsfDkd8tyEtmgkCzZSS2FO4SfxtWw/0XQtcd4Kw/\n5FztMV+pjE6oSnyp5y7x1rCr0ooFcNEk6NavYC+dCVwbdUhsaeQu8fbaH2DWw3D8j2CXIwr60uq5\nS5zpN1fi64NXYcrVMHQkHHlFwV8+nVHPXeJL4S7xtHYJPHAh9KqAM39fsD57Y2m1ZSTGFO4SP5l0\neKHS+hXwxbtbPJ+9KeHIXX8iEk86oSrx89R1sOCf4T6oA/Yr2tto+QGJMw1LJF7enAAv/xZGjIMD\nzi/qW6UDp0xtGYkphbvEx6IZUPVtqDwaTv5p0d8urXnuEmMKd4mHtUvgb2PDfVDPvQtKy4r+lulA\nPXeJL/Xcpe1LbYAJ50PtcrhkSkEvVMpGFzFJnCncpW1zh0cvhZqp4cyYAfu32ltr+QGJM/2bU9q2\n538Ob0+EE66DYa278VdG2+xJjCncpe2a+QA89zPY/3w4qjArPW6P8CIm/YlIPOk3V9qm914IN7je\n5Uj4wi0t2lGpudJa8ldiTOEubc/Hs2DCl6DvbjD6XujQKZIyMuq5S4wp3KVtWbUI7j0XyrrClyZC\nlz6RlRJexKQ/EYknzZaRtqN2OdxzVrj5xsWTofegSMtJB4FG7hJbCndpGzaugXvPgeXvwdiJRV0z\nJl9pzZaRGFO4S/TSG8Me+4dvwHl3w5Bjoq6IIHDc0RWqElsKd4lWJhUu3/ve83DG72Cvz0ddEQCp\nIADQFaoSWxqWSHQyaXjwEnhnEpzyCzhgTNQVbZIJHEA9d4kthbtEI5OGh8fB7EfDFR4PHRd1RVtI\n14e7eu4SV3mFu5mNNLO5ZlZtZldt4/nvmdlsM5tpZk+b2S6FL1USI5OGR74Bbz8IJ/4YDr806oo+\nJZ1RuEu85Qx3MysFbgNOAYYBY8xs2FaHvQ4Md/f9gInA/xS6UEmIdB1MvBjeegBOuBaOKvzG1oWQ\nru+5l2qeu8RUPr+5I4Bqd5/v7nXABGBU4wPc/Vl3r62/+wpQUdgyJRFSG+D+C2BOVdiKOfrKqCtq\nUkPPvUwjd4mpfMJ9ILCw0f2a+seacgnw9209YWbjzGyamU1bunRp/lVK/G1cA3/9Irz7BHz+V22y\nFdNYQ1tGJ1Qlrgo6FdLMxgLDgWO39by7jwfGAwwfPtwL+d7Shq1dGl6g9NFb4XTHNjQrpimbTqhq\nKqTEVD7hvghofB14Rf1jWzCzE4FrgGPdfWNhypPYW7EA7j4TVi+GMffB0JOjrigvmYZ57rqISWIq\nn3CfCuxhZkMIQ300sMW282Z2IPB7YKS7Lyl4lRJPNdPhvtEQpODCKhg0IuqK8pbSbBmJuZzDEndP\nA5cBU4A5wP3uPsvMbjCzhq1xfgF0Bx4wszfMrKpoFUs8zHoY7joVOnaFr0yJVbCDLmKS+Mur5+7u\nk4HJWz12baPvTyxwXRJX7vDPX8IzN8KgQ2H0X1ttQ+tCaui5a8lfiSutLSOFs3FtuJn17Edgn3Ng\n1G1Q1jnqqpolnamf566Ru8SUwl0KY9m8cGXHT+bCSTfAEd+OZGu8QtFsGYk7hbu03OwqePQyKCmB\nsQ/Bbp+NuqIW27z8gNoyEk8Kd2m+1Ab4x3/Ba+Nh54Pg3DuhT2XUVRXEpuUH1JaRmFK4S/MseQce\n+mp4YdLhl8EJ10GHjlFXVTCblh9QW0ZiSuEu2ycI4NXfwVPXQ6fuMGYC7HlK1FUVXErLD0jMKdwl\nfysWQNXl8N4LMPQUOP1W6L5j1FUVRSZQz13iTeEuuQWZcLT+zE/ASuALt8JBX471bJhc0tpmT2JO\n4S7Zffg6TPoefDgD9jgZTvsV9Er+is7arEPiTuEu21a7PLzKdNqd4RWmZ/8R9jk70aP1xrT8gMSd\nwl22lK6D6XfCczfBhlVw6Dfgs1dD515RV9aqtPyAxJ3CXULu4Q5JT10Py+dD5dFwys9hp72jriwS\nmucucadwb+/cofopePa/w/56+V5w/gOwx0ntpgWzLeq5S9wp3NurhlB//n+g5jXoPRhO/y3sPwZK\n9WuxaSqk2jISU/orbm8y6XDVxhdvho/fhp4VcNotcMCXEnWFaUulNu3EpJG7xJPCvb2oXQ4z/gyv\n3QGra6DfnnDG7eHSvAr1T8noClWJOYV7krlDzVSY/md4+0FIr4chx8Cpv4ChI8NVHGWbNi35q3CX\nmFK4J9Gaj+HtifD6PbBkNpR1g/3PgxFfh52GRV1dLKSDgNISw9rxSWWJN4V7UmxYDe8+ATPvh3nP\ngGfCZXi/8Ovw4qNOPaKuMFbSgaslI7GmcI+z2uXw7ydh9qNQ/TRkNoYnSI+6AvY7D8r3jLrC2Mpk\nnDKFu8SYwj1O3MM2S/VT8O4U+OBl8AB67AyHXALDzoCKQ9RLLwCN3CXuFO5t3apFsOCf4TK7856B\nNYvDx3fcG46+MlxLfcCBCvQCSweB5rhLrCnc25IggGX/hg9eqb+9DCveC5/r0geGHAu7nwC7nQC9\nBkZba8JlAtdMGYk1hXtU3MPgXjwTFr8Ji6aHl/9vXB0+37UvDDoMRnwtXOdlp300Om9FqYzCXeJN\n4V5s7rBuKSydC0vfCXvmH88OvzYEeUmHcIGufc+BgQfDoEOh7+7tem2XqGUCp1QbdUiMKdwLwR3W\nfQIr3w+3ols+H5bNg+XzYFk1rF+x+dhOvcK55vueCwP2gwH7Q/lnoKxzZOXLp6UDp0z/UpIYU7jn\nEgSwfnl4InPNx7DmQ1j9IaxeFJ7sXLUQVtVAqnbLn+tZAX13DWewlO8J/YaGX3sO1Ig8BtKZQLNl\nJNbyCnczGwn8GigF7nD3m7Z6vhPwF+BgYBlwnrsvKGypBRAEULcWNqyE9SvDEfX6FWF41y6H2mXh\nCLz2k7CVsnZp+H2Q3uqFLNwYuufO4RK5u58EvQdBnyHQpzJcYbFj1yg+oRSIpkJK3OUMdzMrBW4D\nTgJqgKlmVuXusxsddgmwwt13N7PRwM+B84pRMCsWhP3runXhaLmuFlLrwq9166BuDWxcG4b4xjX1\nt9XhrkIb14TzwptS1g269YWu/cIR9oADwhDvvlN46zEAevQPv2qxrUTLBK5dmCTW8hm5jwCq3X0+\ngJlNAEYBjcN9FHB9/fcTgd+ambm7F7BWAFJvPUzZM9d/6nHHoKwr3rF7/a0b3qkn9BiE9+uBd+q5\n+da596Zb0Lk33mUHvEsf6JBn37sOqKsr6OeStmV9XUYjd4m1fMJ9ILCw0f0a4NCmjnH3tJmtAvoC\nnxSiyMYmbDiMBzbeyHo6hTfvyDo6s4GOsGF7/xhr628fFrpMSYDDdt0h6hJEmq1VT6ia2ThgHMDg\nwYOb9RoHDBtGulv/QpYlsk0jhijcJb7yCfdFwKBG9yvqH9vWMTVm1gHoRXhidQvuPh4YDzB8+PBm\ntWz2rejFvhW9mvOjIiLtRj5njKYCe5jZEDPrCIwGqrY6pgq4sP77c4BnitFvFxGR/OQcudf30C8D\nphBOhfyTu88ysxuAae5eBfwRuNvMqoHlhP8DEBGRiOTVc3f3ycDkrR67ttH3G4BzC1uaiIg0lyby\niogkkMJdRCSBFO4iIgmkcBcRSSCFu4hIAincRUQSSOEuIpJACncRkQRSuIuIJJDCXUQkgRTuIiIJ\npHAXEUkghbuISAJZVMuum9lS4P1I3rxl+lGE7QNjoD1+bn3m9iNOn3sXdy/PdVBk4R5XZjbN3YdH\nXUdra4+fW5+5/Uji51ZbRkQkgRTuIiIJpHDffuOjLiAi7fFz6zO3H4n73Oq5i4gkkEbuIiIJpHBv\nATO70szczPpFXUuxmdkvzOwdM5tpZg+bWe+oayomMxtpZnPNrNrMroq6nmIzs0Fm9qyZzTazWWb2\nnahrai1mVmpmr5vZpKhrKSSFezOZ2SDgc8AHUdfSSv4B7OPu+wHvAldHXE/RmFkpcBtwCjAMGGNm\nw6KtqujSwJXuPgw4DLi0HXzmBt8B5kRdRKEp3JvvZuAHQLs4aeHuT7p7uv7uK0BFlPUU2Qig2t3n\nu3sdMAEYFXFNReXui919Rv33awjDbmC0VRWfmVUAnwfuiLqWQlO4N4OZjQIWufubUdcSka8Af4+6\niCIaCCxsdL+GdhB0DcysEjgQeDXaSlrFLYSDtCDqQgqtQ9QFtFVm9hTQfxtPXQP8kLAlkyjZPrO7\nP1p/zDWE/4S/tzVrk9ZhZt2BB4Er3H111PUUk5mdBixx9+lmdlzU9RSawr0J7n7ith43s32BIcCb\nZgZhe2KGmY1w949ascSCa+ozNzCzi4DTgBM82XNoFwGDGt2vqH8s0cysjDDY73X3h6KupxUcCZxu\nZqcCnYGeZnaPu4+NuK6C0Dz3FjKzBcBwd4/LokPNYmYjgV8Bx7r70qjrKSYz60B40vgEwlCfCpzv\n7rMiLayILByp/BlY7u5XRF1Pa6sfuX/f3U+LupZCUc9d8vVboAfwDzN7w8x+F3VBxVJ/4vgyYArh\nicX7kxzs9Y4ELgCOr//v+0b9iFZiSiN3EZEE0shdRCSBFO4iIgmkcBcRSSCFu4hIAincRUQSSOEu\nIpJACncRkQRSuIuIJND/AyGL2ZQnM06qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11261c3c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy  as np\n",
    "%matplotlib inline\n",
    "\n",
    "def step_function(x):\n",
    "    return np.array(x>0, dtype=np.int)\n",
    "\n",
    "def sigmoid_function(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "x = np.arange(-5, 5, 0.1)\n",
    "y_step = step_function(x)\n",
    "\n",
    "y_sigmoid = sigmoid_function(x)\n",
    "\n",
    "plt.plot(x, y_step)\n",
    "plt.plot(x, y_sigmoid)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### ReLU関数 ( Rectified Liner Unit)\n",
    "\n",
    "$\n",
    "h(x) = \n",
    "\\begin{eqnarray}\n",
    "\\left\\{\n",
    "\\begin{array}{l}\n",
    "0 & ( x \\leqq 0 ) \\\\\n",
    "x & ( x > 0 ) \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\end{eqnarray}\n",
    "$\n",
    "\n",
    "* 最近ニューラルネットワークでよく使われる関数\n",
    "    * 0以下のとき\n",
    "        * 0を出力\n",
    "    * 0を超えているとき\n",
    "        * 入力をそのまま出力\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGdhJREFUeJzt3XtU1HXeB/D3R+SigASCYiCghoGKQrBe1mxL3VK7WD3a\nqtturVtutRK2ppv2dN3d2ic7pWm7+7jVbrUJmpfsqptlm24XUy4Kgop3UBQUUZT7fJ4/pB5tUQac\nme/8Zt6vczyCzhneczy85+NnfnxHVBVERGQdHUwHICKitmFxExFZDIubiMhiWNxERBbD4iYishgW\nNxGRxbC4iYgshsVNRGQxLG4iIovp6Iw7DQ8P17i4OGfcNRGRR9qyZUuFqkbYc1u7iltE9gE4BaAJ\nQKOqpl3s9nFxcdi8ebM9d01ERABEZL+9t23LxH2dqla0Iw8RETkQd9xERBZjb3ErgH+KyBYRmebM\nQEREdHH2rkquVtVSEekG4GMRKVLVz8+9QXOhTwOAmJgYB8ckIqJv2TVxq2pp8+9HAawCMLiF2yxW\n1TRVTYuIsOuFUSIiaodWi1tEAkUk+NuPAVwPIN/ZwYiIrORMfSMqT9e75GvZM3F3B7BRRPIAbALw\ngaqucW4sIiJrefLdAty0cCNO1zU6/Wu1uuNW1T0ABjk9CRGRRa3OLcWyzSWYft0VCPR3ys81noeX\nAxIRXYJ9Fafx6Kp8pMWGYsboeJd8TRY3EVE71TU2IT0zBz4dBAsmp6Cjj2sq1fkzPRGRh3puzQ5s\nK63C//4sFVGXdXLZ1+XETUTUDp8UHsGrG/fi58NicUP/SJd+bRY3EVEbHa6qwcNv5yGxRxfMHZfo\n8q/P4iYiaoMmmyIjKxd1jTYsmpKCAF8fl2fgjpuIqA0WfroLm/Yex/MTB6FPRJCRDJy4iYjs9NWe\nY3jpk124LSUKE1KjjeVgcRMR2eH46XrMyMpFbNdA/O7WAUazcFVCRNQKVcWst/Nw/HQ9Vt71QwS5\n4KcjL4YTNxFRK/727334pOgo5o5LwICoENNxWNxERBezraQKz35UiNGJ3XHXD+NMxwHA4iYiuqDq\nukakZ2YjPMgf8yYMhIiYjgSAO24iohapKh5dtQ0Hjp9B1rRhCA30Mx3pO5y4iYhasHxLCVbnHsJD\no/ticK8w03HOw+ImIvqe4qPVeHx1AYb17ooHrrvCdJz/wOImIjpHbUMTpi/JRic/H8yflAyfDu6x\n1z4Xd9xEROf4wweFKCo7hb/d/QN07xJgOk6LOHETETVbk38Yb361H/eO6IXrErqZjnNBLG4iIgAl\nlWcwe/lWDIwOwawbEkzHuSgWNxF5vYYmGx7MzIFNgYWTU+DX0b2rkTtuIvJ6L368E9kHTuClySmI\n7RpoOk6r3PtphYjIyTbsKsef/7Ubk37QE7cMutx0HLuwuInIa5WfqsNDS/PQJyIIT9zc33Qcu3FV\nQkReyWZT/GZZLk7VNuAf9wxGJz/XvwVZe3HiJiKvtHjDHmzYVYEnbu6PhMgupuO0CYubiLxO9oFK\nPL92B25M6oHJg3uajtNmLG4i8ipVNQ14MDMHkSEBeOb2JLc5qrUtuOMmIq+hqnhkxVaUVdXi7fuG\nIaSTr+lI7cKJm4i8xltfH8BH+WWYdcOVSIkJNR2n3VjcROQVCg+fxNPvb8c1fSNw74jepuNcEhY3\nEXm8M/WNSM/MQUgnX7xwxyB0cMOjWtuCO24i8nhPvbsdu8ur8ebUIQgP8jcd55LZPXGLiI+I5IjI\n+84MRETkSKtzS7F080E8cG0fXB0fbjqOQ7RlVZIBoNBZQYiIHG1fxWk8uiofqbGheGh0X9NxHMau\n4haRaAA3AnjFuXGIiByjvtGG9Mwc+HQQvDQ5BR19POclPXsfyXwAswHYLnQDEZkmIptFZHN5eblD\nwhERtddza4qwrbQKz00YiKjLOpmO41CtFreI3ATgqKpuudjtVHWxqqapalpERITDAhIRtdWnRUfw\nysa9uGtYLG7oH2k6jsPZM3EPB3CLiOwDkAVgpIj8w6mpiIjaqayqFjOX5SGxRxfMGZdoOo5TtFrc\nqjpHVaNVNQ7AJACfquqdTk9GRNRGTTZFRlYO6hptWDQlBQG+1jmqtS14HTcReYxFnxbj673HMW/C\nQPSJCDIdx2naVNyq+hmAz5yShIjoEny95xgWfLITt6VEYUJqtOk4TuU518cQkdeqPF2PjKxcxHYN\nxO9uHWDJo1rbgqsSIrI0VcWs5Xk4froeK+/6IYL8Pb/WOHETkaX97d/7sK7wKOaMS8CAqBDTcVyC\nxU1ElrWtpArPflSI0YndcfcP40zHcRkWNxFZUnVdI9IzsxEe5I95EwZ6/F77XJ6/DCIij6Oq+O9V\n23Dg+BlkTRuG0EA/05FcihM3EVnO8i0leCf3EGaM7ovBvcJMx3E5FjcRWUrx0Wo8vroAQ3uH4dfX\nXWE6jhEsbiKyjNqGJkxfko1Ofj5YMCkFPhZ/C7L24o6biCzjmQ8LUVR2Cn+7+wfo3iXAdBxjOHET\nkSWsyS/DG1/ux70jeuG6hG6m4xjF4iYit1dSeQazl+dhYHQIZt2QYDqOcSxuInJrDU02ZGTlwqbA\nwskp8OvI2uKOm4jc2vx1O7FlfyUWTk5BbNdA03HcAp+6iMhtbdxVgT99thuTftATNw+63HQct8Hi\nJiK3VH6qDjOW5uKKiCA8cXN/03HcClclROR2bDbFb5bl4lRtA966Zwg6+XnmW5C1FyduInI7izfs\nwYZdFXji5v64MjLYdBy3w+ImIreSfaASz6/dgRuTemDy4J6m47glFjcRuY2qmgY8mJmDyJAAPHN7\nklcd1doW3HETkVtQVTyyYivKqmqx7L5hCOnkazqS2+LETURuYcmmA/govwwP33AlrooJNR3HrbG4\nici4orKTePq97bimbwSmjehtOo7bY3ETkVE19U1IX5KDLp188cIdg9DBS49qbQvuuInIqCffLUBx\neTXenDoE4UH+puNYAiduIjJmdW4plm4+iAeu7YOr48NNx7EMFjcRGbH/2Gk8uiofabGheGh0X9Nx\nLIXFTUQuV99oQ3pmDjoIsGByCjr6sIragjtuInK5eWuLsLWkCn+5MxVRl3UyHcdy+DRHRC71adER\n/HXDXvx8WCzGDIg0HceSWNxE5DJlVbV4+O2tSOzRBXPHJZqOY1ksbiJyiSabYsbSHNQ2NGHRlBQE\n+PKo1vbijpuIXOLl9cX4as9xzJswEH0igkzHsbRWJ24RCRCRTSKSJyIFIvKUK4IRkefYtPc45q/b\nidtSojAhNdp0HMuzZ+KuAzBSVatFxBfARhH5SFW/cnI2IvIAlafrkZGVg5iwzvjdrQN4VKsDtFrc\nqqoAqps/9W3+pc4MRUSeQVUxa3keKqrrsOqB4Qjy53bWEex6cVJEfEQkF8BRAB+r6tct3GaaiGwW\nkc3l5eWOzklEFvT3L/ZhXeFRzBmbiAFRIabjeAy7iltVm1Q1GUA0gMEiMqCF2yxW1TRVTYuIiHB0\nTiKymPzSKjz7YRFGJ3bDL4bHmY7jUdp0OaCqngCwHsAY58QhIk9QXdeI6UuyERboh3kTBnGv7WD2\nXFUSISKXNX/cCcCPARQ5OxgRWZOq4rF38nHg+BksmJSM0EA/05E8jj2vFPQA8LqI+OBs0S9T1fed\nG4uIrGpFdilW5ZTiodF9MaR3V9NxPJI9V5VsBZDigixEZHG7y6vx2Dv5GNIrDNNHXmE6jsfij7wT\nkUPUNjRh+pIcdPLzwYJJKfDhW5A5DS+qJCKHeObDQhQePonX7k5DZEiA6TgejRM3EV2yNflleOPL\n/bjn6l4YmdDddByPx+ImoktSUnkGs5fnISkqBLPHJJiO4xVY3ETUbo1NNmRk5cKmwKIpKfDryEpx\nBe64iajdXly3E1v2V+KlySmI7RpoOo7X4NMjEbXLxl0V+NNnu/GTtJ64ZdDlpuN4FRY3EbVZ+ak6\nPLQsF73DA/HELf1Mx/E6XJUQUZvYbIrfLMtFVU0D3vzlYHT2Y424GiduImqTxRv2YMOuCjx+Uz8k\nRHYxHccrsbiJyG7ZByrx/NodGJcUiZ8OiTEdx2uxuInILlU1DXgwMwfduwTg2dsH8qhWg7icIqJW\nqSrmrtyGsqpaLLtvGEI6+ZqO5NU4cRNRq5ZsOoAPth3GwzdciatiQk3H8XosbiK6qKKyk3j6ve0Y\nER+OaSN6m45DYHET0UWcqW/E9CU5CA7wxQt3JKMDj2p1C9xxE9EFPf3eduwur8abU4cgItjfdBxq\nxombiFr0bt4hZH1zEA9c2wdXx4ebjkPnYHET0X84cOwM5q7chtTYUMwY3dd0HPoeFjcRnae+0Yb0\nzGx0EGDBpGT4+rAm3A133ER0nnlri5BXUoW/3HkVokM7m45DLeBTKRF9Z33RUfx1w178fFgsxgzo\nYToOXQCLm4gAAEdO1mLm23lIiAzG3HGJpuPQRbC4iQhNNsWMrFzU1Ddh0ZSrEODrYzoSXQR33ESE\nl9cX48s9x/DchIG4oluQ6TjUCk7cRF5u097jmL9uJ25NvhwTU6NNxyE7sLiJvFjl6XpkZOUgJqwz\nfn9bEo9qtQiuSoi8lKpi1vI8VFTXYeX9wxHkzzqwCk7cRF7q71/sw7rCo3hkbCKSokNMx6E2YHET\neaH80io8+2ERRiV0w9ThcabjUBuxuIm8THVdI9IzcxAW6Id5Ewdxr21BXGoReZnH38nH/mOnseTe\noQgL9DMdh9qh1YlbRHqKyHoR2S4iBSKS4YpgROR4y7eUYGVOKR4cFY+hvbuajkPtZM/E3Qhgpqpm\ni0gwgC0i8rGqbndyNiJyoN3l1XjsnXwM6RWG9JHxpuPQJWh14lbVw6qa3fzxKQCFAKKcHYyIHKe2\noQnTl+QgwLcDFkxKgQ/fgszS2rTjFpE4ACkAvnZGGCJyjmc/LETh4ZN47e40RIYEmI5Dl8juq0pE\nJAjACgAzVPVkC38/TUQ2i8jm8vJyR2YkokuwJr8Mr3+5H1OH98LIhO6m45AD2FXcIuKLs6X9lqqu\nbOk2qrpYVdNUNS0iIsKRGYmonUoqz2D28jwkRYXgt2OvNB2HHMSeq0oEwKsAClX1BedHIiJHaGyy\nISMrFzYFFk5OgX9HHtXqKeyZuIcD+BmAkSKS2/xrnJNzEdElmr9uF7bsr8QfbhuAuPBA03HIgVp9\ncVJVNwLgS9BEFvLv4gq8/FkxJqZGY3wyLwLzNPyRdyIPU36qDjOW5qJ3eCCeGt/fdBxyAv7IO5EH\nsdkUM9/OQ1VNA96YOhid/fgt7ok4cRN5kL9u2IPPd5bjsZv6IbFHF9NxyElY3EQeIudAJeat3YGx\nAyJx55AY03HIiVjcRB6gqqYB6Zk56N4lAH+8fSCPavVwXIARWZyqYu7KbThcVYtlvxqGkM6+piOR\nk3HiJrK4zE0H8cG2w5h5fV+kxoaajkMuwOImsrAdZafw1HsFGBEfjvuu6WM6DrkIi5vIomrqmzB9\nSTaCA3zxwh3J6MCjWr0Gd9xEFvX0+wUoLq/GG1MHIyLY33QcciFO3EQW9F7eIWRuOoj7f9QHI+J5\nGqe3YXETWcyBY2cwd+U2pMaG4qEf9zUdhwxgcRNZSH2jDemZ2RABFkxKhq8Pv4W9EXfcRBYyb20R\n8kqq8Jc7r0J0aGfTccgQPl0TWcT6oqP464a9+NnQWIwZ0MN0HDKIxU1kAUdO1mLm23lIiAzGozcm\nmo5DhrG4idxck00xIysXNfVNWDTlKgT48i3IvB133ERu7k/ri/HlnmOYN2EgrugWZDoOuQFO3ERu\nbNPe43hx3U7cmnw5JqRGm45DboLFTeSmKk/XIyMrBzFhnfH725J4VCt9h6sSIjekqpi1fCsqquuw\n8v7hCPLntyr9P07cRG7o9S/2YV3hEcwZm4ik6BDTccjNsLiJ3Ex+aRWe+bAIoxK64RfD40zHITfE\n4iZyI9V1jUjPzEFYoB/mTRzEvTa1iIszIjfy+Dv52H/sNJbcOxRhgX6m45Cb4sRN5CZWbCnBypxS\nPDgqHkN7dzUdh9wYi5vIDewur8Zjq/MxpFcY0kfGm45Dbo7FTWRYbUMT0pfkwL9jByyYlAIfvgUZ\ntYI7biLDnv2wENsPn8Srd6UhMiTAdByyAE7cRAatLSjD61/uxz1X98KoxO6m45BFsLiJDCk9UYPZ\ny7ciKSoEs8ckmI5DFsLiJjKgscmGjMwcNNkUCyenwK8jvxXJftxxExkwf90ubN5fiQWTkhEXHmg6\nDllMq0/zIvKaiBwVkXxXBCLydP8ursDLnxVjYmo0xidHmY5DFmTP/8/+DmCMk3MQeYWK6jrMWJqL\nPhFBeGp8f9NxyKJaLW5V/RzAcRdkIfJoNpti5rI8VNU0YNGUFHT246aS2oeviBC5yCsb9+BfO8vx\n+E39kBDZxXQcsjCHFbeITBORzSKyuby83FF3S+QRcg+ewHNrdmDsgEj8dEiM6ThkcQ4rblVdrKpp\nqpoWERHhqLslsryTtQ1Iz8xG9y4B+OPtA3lUK10yLtmInEhVMWfFNhw6UYtlvxqGkM6+piORB7Dn\ncsBMAF8CuFJESkTkl86PReQZsr45iA+2HcbD11+J1NhQ03HIQ7Q6cavqZFcEIfI0O4+cwpPvFmBE\nfDh+dU1v03HIg/CqEiInqKlvwvQl2QgO8MULdySjA49qJQfijpvICZ5+vwA7j1TjzV8ORkSwv+k4\n5GE4cRM52PtbDyFz00Hcf20fjIjnFVbkeCxuIgc6cOwM5qzYhqtiLsNvftzXdBzyUCxuIgepb7Qh\nPSsHIsCCSSnw9eG3FzkHd9xEDvL8P3cg7+AJ/PmnV6FnWGfTcciDcSQgcoD1O45i8ed7cOfQGIxN\n6mE6Dnk4FjfRJTpyshYzl+UhITIY/31jP9NxyAuwuIkuQZNN8dDSXNTUN2HRlBQE+PqYjkRegDtu\nokvw58+K8cXuY3huwkBc0S3YdBzyEpy4idrpm33H8eK6XRiffDkmpkabjkNehMVN1A4nztQjIzMH\n0aGd8PtbB/CoVnIprkqI2khVMWv5VpRX12Hl/cMRHMCjWsm1OHETtdEbX+7Hx9uP4JGxiUiKDjEd\nh7wQi5uoDQoOVeEPHxRiVEI3TB0eZzoOeSkWN5GdTtc1In1JDkIDfTFv4iDutckY7riJ7PTY6nzs\nO3Yab90zFGGBfqbjkBfjxE1khxVbSrAyuxTpI+MxrE9X03HIy7G4iVqxp7waj63Ox5BeYXhwVLzp\nOEQsbqKLqWtsQnpmDvw7dsD8Scnw4VuQkRvgjpvoIp79sAgFh07ilZ+noUdIJ9NxiABw4ia6oH8W\nlOHvX+zD1OG9MLpfd9NxiL7D4iZqwaETNZi1fCuSokLw27FXmo5DdB4WN9H3NDbZkJGVg8YmGxZO\nToF/Rx7VSu6FO26i73npk134Zl8l5v8kGXHhgabjEP0HTtxE5/iiuAIL1xdjYmo0bk2JMh2HqEUs\nbqJmFdV1yFiai97hgXhqfH/TcYguiKsSIgA2m+Lht/NQVdOAN6YORmc/fmuQ++LETQTg1Y178dmO\ncjx2YyISe3QxHYfooljc5PVyD57A/6wpwtgBkbhzaKzpOEStYnGTVztZ24D0zGx07xKAP94+kEe1\nkiVwkUdeS1UxZ+U2HDpRi2W/GoaQznwLMrIGTtzktbK+OYgPth7GzOv7IjU21HQcIrvZVdwiMkZE\ndohIsYg84uxQRM6288gpPPluAUbEh+O+a/qYjkPUJq0Wt4j4AHgZwFgA/QBMFpF+zg5G5Cw19U2Y\nviQbwQG+eOGOZHTgUa1kMfbsuAcDKFbVPQAgIlkAxgPY7ugwB4+fgaqj75XofC+vL8bOI9V485eD\nERHsbzoOUZvZU9xRAA6e83kJgCHOCHP9i5+jpqHJGXdNdJ77r+2DEfERpmMQtYvDrioRkWkApgFA\nTExMu+7jj/+VhMYmjtzkXMEBHTEyoZvpGETtZk9xlwLoec7n0c1/dh5VXQxgMQCkpaW1q33HJ/NQ\nHyKi1thzVck3AOJFpJeI+AGYBOBd58YiIqILaXXiVtVGEZkOYC0AHwCvqWqB05MREVGLRJ1wGYeI\nlAPY7/A7dr5wABWmQ7iYNz5mwDsfNx+ze4tVVbteMXdKcVuViGxW1TTTOVzJGx8z4J2Pm4/Zc/BH\n3omILIbFTURkMSzu8y02HcAAb3zMgHc+bj5mD8EdNxGRxXDiJiKyGBZ3C0RkpoioiISbzuIKIjJP\nRIpEZKuIrBKRy0xnchZvPKJYRHqKyHoR2S4iBSKSYTqTq4iIj4jkiMj7prM4Eov7e0SkJ4DrARww\nncWFPgYwQFUHAtgJYI7hPE7hxUcUNwKYqar9AAwF8GsvedwAkAGg0HQIR2Nx/6cXAcwG4DXLf1X9\np6o2Nn/6Fc6eR+OJvjuiWFXrAXx7RLFHU9XDqprd/PEpnC0yjz8YSESiAdwI4BXTWRyNxX0OERkP\noFRV80xnMWgqgI9Mh3CSlo4o9vgCO5eIxAFIAfC12SQuMR9nhzCb6SCO5nVvFiwi6wBEtvBXjwKY\ni7NrEo9zscetqqubb/Mozv63+i1XZiPXEJEgACsAzFDVk6bzOJOI3ATgqKpuEZFrTedxNK8rblUd\n3dKfi0gSgF4A8kQEOLsuyBaRwapa5sKITnGhx/0tEbkbwE0ARqnnXiNq1xHFnkhEfHG2tN9S1ZWm\n87jAcAC3iMg4AAEAuojIP1T1TsO5HILXcV+AiOwDkKaqVjmgpt1EZAyAFwD8SFXLTedxFhHpiLMv\nvo7C2cL+BsAUTz/tUs5OIq8DOK6qM0zncbXmifthVb3JdBZH4Y6bAGARgGAAH4tIroj8xXQgZ2h+\nAfbbI4oLASzz9NJuNhzAzwCMbP73zW2eRMmiOHETEVkMJ24iIothcRMRWQyLm4jIYljcREQWw+Im\nIrIYFjcRkcWwuImILIbFTURkMf8HVLFUJYC057YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11287c048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy  as np\n",
    "%matplotlib inline\n",
    "\n",
    "def relu_function(x):\n",
    "    return np.maximum(0, x)\n",
    "\n",
    "x = np.arange(-5, 5, 0.1)\n",
    "y = relu_function(x)\n",
    "\n",
    "plt.plot(x, y)\n",
    "plt.ylim(-0.1, 5.1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 多次元配列の計算\n",
    "\n",
    "* 対応\n",
    "    * 1次元配列 : ベクトル\n",
    "    * 2次元配列 : 行列\n",
    "\n",
    "### ベクトル (1次元要素数4)\n",
    "\n",
    "$\n",
    "  \\boldsymbol{A} = (1, 2, 3, 4)\n",
    "$\n",
    "\n",
    "### 行列 (3 x 2)\n",
    "\n",
    "$\n",
    "  \\boldsymbol{B} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      1 & 2 \\\\\n",
    "      3 & 4 \\\\\n",
    "      5 & 6\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "$\n",
    "\n",
    "### 行列 (2 x 2)\n",
    "\n",
    "$\n",
    "  \\boldsymbol{C} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      c_{11} & c_{12} \\\\\n",
    "      c_{21} & c_{22}\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "$\n",
    "\n",
    "Pythonで表現すると\n",
    "\n",
    "```\n",
    "A = np.array([1,2,3,4])\n",
    "```\n",
    "\n",
    "```\n",
    "### 3 x 2 の配列\n",
    "### 最初の次元に3つの要素があり、次の次元に2つの要素がある\n",
    "B = np.array([[1,2], [3,4], [5,6]])\n",
    "```\n",
    "\n",
    "```\n",
    "### 2 x 2 の配列\n",
    "### 最初の次元に2つの要素があり、次の次元に2つの要素がある\n",
    "C = np.array([[c11,c12], [c21,c22]])\n",
    "```\n",
    "\n",
    "* 行列(matrix)\n",
    "    * 行(row)\n",
    "        * 配列の横方向の並び\n",
    "    * 列(column)\n",
    "        * 配列の横方向の並び\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3 4]\n",
      "[[1 2]\n",
      " [3 4]\n",
      " [5 6]]\n",
      "--- 次元 ---\n",
      "1\n",
      "2\n",
      "--- 4x1の配列(1次元要素数4) ---\n",
      "(4,)\n",
      "--- 3x2の配列(2次元要素数3) ---\n",
      "(3, 2)\n"
     ]
    }
   ],
   "source": [
    "import numpy  as np\n",
    "\n",
    "### 1次元\n",
    "A = np.array([1,2,3,4])\n",
    "print(A)\n",
    "\n",
    "### 2次元\n",
    "B = np.array([[1,2], [3,4], [5,6]])\n",
    "print(B)\n",
    "\n",
    "### 配列の次元数\n",
    "print(\"--- 次元 ---\")\n",
    "print(np.ndim(A))\n",
    "print(np.ndim(B))\n",
    "\n",
    "### 配列の形状\n",
    "print (\"--- 4x1の配列(1次元要素数4) ---\")\n",
    "print(A.shape)\n",
    "print (\"--- 3x2の配列(2次元要素数3) ---\")\n",
    "print(B.shape)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 行列の内積\n",
    "\n",
    "$\n",
    "  \\boldsymbol{A} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      a_{11} & a_{12} \\\\\n",
    "      a_{21} & a_{22}\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  \\boldsymbol{B} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      b_{11} & b_{12} \\\\\n",
    "      b_{21} & b_{22}\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  \\boldsymbol{A} \\cdot  \\boldsymbol{B} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      a_{11}b_{11} + a_{12}b_{21} & a_{11}b_{12} + a_{12}b_{22}  \\\\\n",
    "      a_{21}b_{11} + a_{22}b_{21} & a_{21}b_{12} + a_{22}b_{22}\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 2)\n",
      "(2, 2)\n",
      "[[19 22]\n",
      " [43 50]]\n",
      "(3, 2)\n",
      "(2, 3)\n",
      "[[ 27  30  33]\n",
      " [ 61  68  75]\n",
      " [ 95 106 117]]\n",
      "(3, 3)\n"
     ]
    }
   ],
   "source": [
    "import numpy  as np\n",
    "\n",
    "A = np.array([[1,2],[3,4]])\n",
    "B = np.array([[5,6],[7,8]])\n",
    "\n",
    "### 配列の形状\n",
    "print(A.shape)\n",
    "print(B.shape)\n",
    "\n",
    "### 内積の計算\n",
    "print(np.dot(A,B))\n",
    "\n",
    "C = np.array([[1,2],[3,4],[5,6]])\n",
    "D = np.array([[7,8,9],[10,11,12]])\n",
    "\n",
    "### 配列の形状\n",
    "print(C.shape)\n",
    "print(D.shape)\n",
    "\n",
    "### 内積の計算\n",
    "print(np.dot(C,D))\n",
    "\n",
    "### 3 x 2と 2 x 3の行列の内積は 3 x 3になる\n",
    "print(np.dot(C,D).shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## ニューラルネットワークの内積\n",
    "\n",
    "### 重みだけがあるニューラルネットワークの実装\n",
    "\n",
    "* バイアスと活性化関数は省略\n",
    "    * 重みだけある\n",
    "\n",
    "<img src=\"img/3-14.png\" alt=\"ニューラルネットワークの計算を行列の積で行う\" title=\"ニューラルネットワークの計算を行列の積で行う\" width=\"300\" height=\"300\" />\n",
    "\n",
    "$\n",
    "  \\boldsymbol{X} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      1 & 2\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  \\boldsymbol{W} = \\left(\n",
    "    \\begin{array}{ccc}\n",
    "      1 & 3 & 5 \\\\\n",
    "      2 & 4 & 6\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  \\boldsymbol{Y} = \\boldsymbol{X} \\cdot  \\boldsymbol{W} = \\left(\n",
    "    \\begin{array}{ccc}\n",
    "      5 & 11 & 17\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2,)\n",
      "(2, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 5, 11, 17])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy  as np\n",
    "\n",
    "X = np.array([1,2])\n",
    "W = np.array([[1,3,5],[2,4,6]])\n",
    "\n",
    "print(X.shape)\n",
    "print(W.shape)\n",
    "\n",
    "np.dot(X,W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3層ニューラルネットワークの実装\n",
    "\n",
    "<img src=\"img/3-15.png\" alt=\"3層ニューラルネットワーク\" title=\"3層ニューラルネットワーク\" width=\"400\" height=\"400\" />\n",
    "\n",
    "* 3層ニューラルネットワーク\n",
    "    * 入力層(第0層) : ニューロン2つ\n",
    "    * 1つ目隠れ層(第1層) : ニューロン3つ\n",
    "    * 2つ目隠れ層(第2層) : ニューロン2つ\n",
    "    * 出力層(第3層) : ニューロン2つ\n",
    "\n",
    "<img src=\"img/3-17.png\" alt=\"入力層から第一層目への信号の伝達\" title=\"入力層から第一層目への信号の伝達\" width=\"400\" height=\"400\" />\n",
    "\n",
    "* 記号の定義\n",
    "    * 入力層(第0層) $ x_2 $ のニューロンから次層（第1層）のニューロン $ a_1^{(1)} $ への重み $ w_{12}^{(1)}$\n",
    "\n",
    "$\n",
    "\\displaystyle a_{1\\text{:1番目のニューロン}}^{(1)\\text{:第一層のニューロン}}\n",
    "$\n",
    "\n",
    "$\n",
    "\\displaystyle w_{12\\text{:次層の1番目のニューロン、前層の2番目のニューロン}}^{(1)\\text{:第一層目の重み}}\n",
    "$\n",
    "\n",
    "* バイアスのニューロンのインデックスが一つ\n",
    "    * ひとつしかバイアスのニューロンが無いため\n",
    "\n",
    "上図の時、 \n",
    "\n",
    "$\n",
    "a_1^{(1)} = w_{11}^{(1)} x_1 + w_{12}^{(1)} x_2 + b_{1}^{(1)}\n",
    "$\n",
    "\n",
    "と重み付き信号とバイアスの和で表現される。また、行列の内積を用いると、第一層目の「重み付き和」は\n",
    "\n",
    "$\n",
    "\\boldsymbol{A}^{(1)} = \\boldsymbol{X} \\boldsymbol{W}^{(1)} + \\boldsymbol{B}^{(1)}\n",
    "$\n",
    "\n",
    "$\n",
    "  \\boldsymbol{A}^{(1)} = \\left(\n",
    "    \\begin{array}{ccc}\n",
    "      a^{(1)}_1 & a^{(1)}_2 & a^{(1)}_3\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  ,\n",
    "  \\boldsymbol{X} = \\left(\n",
    "    \\begin{array}{cc}\n",
    "      x_1 & x_2\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  ,\n",
    "  \\boldsymbol{B}^{(1)} = \\left(\n",
    "    \\begin{array}{ccc}\n",
    "      b^{(1)}_1 & b^{(1)}_2 & b^{(1)}_3\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "  ,\n",
    "  \\boldsymbol{W}^{(1)} = \\left(\n",
    "    \\begin{array}{ccc}\n",
    "      w^{(1)}_{11} & w^{(1)}_{21} & w^{(1)}_{31} \\\\\n",
    "      w^{(1)}_{12} & w^{(1)}_{22} & w^{(1)}_{32}\n",
    "    \\end{array}\n",
    "  \\right)\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 第一層"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n",
      "(2,)\n",
      "(3,)\n",
      "[ 0.3  0.7  1.1]\n",
      "[ 0.57444252  0.66818777  0.75026011]\n"
     ]
    }
   ],
   "source": [
    "import numpy  as np\n",
    "\n",
    "X = np.array([1.0, 0.5])\n",
    "W1 = np.array([[0.1, 0.3, 0.5], [0.2, 0.4, 0.6]])\n",
    "B1 = np.array([0.1,0.2,0.3])\n",
    "\n",
    "print(W1.shape)\n",
    "print(X.shape)\n",
    "print(B1.shape)\n",
    "\n",
    "A1 = np.dot(X, W1) + B1\n",
    "print(A1)\n",
    "\n",
    "def sigmoid_function(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "Z1 = sigmoid_function(A1)\n",
    "print(Z1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 第二層"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3,)\n",
      "(3, 2)\n",
      "(2,)\n",
      "[ 0.51615984  1.21402696]\n",
      "[ 0.62624937  0.7710107 ]\n"
     ]
    }
   ],
   "source": [
    "W2 = np.array([[0.1, 0.4], [0.2, 0.5], [0.3, 0.6]])\n",
    "B2 = np.array([0.1,0.2])\n",
    "print(Z1.shape)\n",
    "print(W2.shape)\n",
    "print(B2.shape)\n",
    "A2 = np.dot(Z1, W2) + B2\n",
    "Z2 = sigmoid_function(A2)\n",
    "print(A2)\n",
    "print(Z2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 第三層(出力層)\n",
    "\n",
    "* identity_function\n",
    "    * 恒等関数(入力をそのまま出力する関数)\n",
    "    * 出力層の活性化関数\n",
    "        * 出力層の活性化関数は解く問題に応じて変化する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.31682708  0.69627909]\n"
     ]
    }
   ],
   "source": [
    "def identity_function(x):\n",
    "    return x\n",
    "\n",
    "W3 = np.array([[0.1, 0.3], [0.2, 0.4]])\n",
    "B3 = np.array([0.1,0.2])\n",
    "A3 = np.dot(Z2, W3) + B3\n",
    "Y = identity_function(A3)\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### まとめ\n",
    "\n",
    "* 記号の慣例\n",
    "    * 重みは「W1」のように大文字\n",
    "    * それ以外(バイアスや中間結果など)は小文字\n",
    "* 下のプログラムについて\n",
    "    * forward\n",
    "        * 入力から出力方向への伝達処理を表す\n",
    "            * バックワード(backward)方向\n",
    "            * 出力から入力方向"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.31682708  0.69627909]\n"
     ]
    }
   ],
   "source": [
    "def sigmoid_function(x):\n",
    "    return 1/(1 + np.exp(-x))\n",
    "\n",
    "def identity_function(x):\n",
    "    return x\n",
    "\n",
    "### 重みとバイアスの初期化\n",
    "def init_network():\n",
    "    network = {}\n",
    "    network['W1'] = np.array([[0.1,0.3,0.5], [0.2,0.4,0.6]])\n",
    "    network['b1'] = np.array([0.1,0.2,0.3])\n",
    "    network['W2'] = np.array([[0.1,0.4], [0.2,0.5],[0.3,0.6]])\n",
    "    network['b2'] = np.array([0.1,0.2])\n",
    "    network['W3'] = np.array([[0.1,0.3], [0.2,0.4]])\n",
    "    network['b3'] = np.array([0.1,0.2])\n",
    "    \n",
    "    return network\n",
    "\n",
    "### 入力信号が出力信号に変換されるプロセスの実装\n",
    "def forward(network, x):\n",
    "    W1, W2, W3 = network['W1'], network['W2'], network['W3']\n",
    "    b1, b2, b3 = network['b1'], network['b2'], network['b3']\n",
    "    \n",
    "    a1 = np.dot(x, W1) + b1\n",
    "    z1 = sigmoid_function(a1)\n",
    "    a2 = np.dot(z1, W2) + b2\n",
    "    z2 = sigmoid_function(a2)\n",
    "    a3 = np.dot(z2, W3) + b3\n",
    "    y = identity_function(a3)\n",
    "    \n",
    "    return y\n",
    "\n",
    "network = init_network()\n",
    "x = np.array([1.0, 0.5])\n",
    "y = forward(network, x)\n",
    "print(y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
